feat(catalog): page and filter the storefront server-side (steps 1-2)
The real catalogue reaches ~16k products and GET /catalog/products had no
paging parameters at all: it returned every active product with its
variants, images, category, brand and collections in one response. The
apps' PAGE_SIZE = 20 was only a render window over that full array, so
scrolling made zero requests. The reference paged this server-side; the
port had moved it to the client (STUB-03), so this is a return to
reference behaviour.
Step 1 — indexes, which were not optional. Product carried NO index
whatsoever (not isActive, categoryId, brandId or createdAt) and
ProductVariant had none on productId, so every per-product include
scanned its whole table. Adding take/skip on top of that would have
paged the response without making the query faster, which is the exact
failure this change exists to avoid. Product_isActive_createdAt_idx is
composite so one index serves the filter, the default ordering and the
OFFSET walk.
CREATE INDEX, not CONCURRENTLY: the deploy image's migration runner
wraps each file in a transaction, which CONCURRENTLY cannot join (D15).
Every statement is IF NOT EXISTS because both migration paths run it.
Step 2 — the routes. Paging is opt-in, which is the whole
backward-compatibility story: with no query parameters the route returns
the same plain array from the same cache as before, so the admin panel,
the smoke scripts and installed app builds are untouched. Pass page,
limit or any filter and it becomes { rows, total, page, limit }.
The filters had to move too, not just the paging. The apps filtered
title, category, brand and gram range in the browser over the full
array; paging the response while leaving that in place would have made
search quietly WRONG — searching only the first page and reporting
"محصولی پیدا نشد" for products that exist. So every filter is built into
the Prisma query, and count uses the same where clause, so `total`
describes the set the app is actually scrolling.
Three routes added so the apps can stop fetching everything to find one
thing (wired up in step 3): ?ids=a,b,c for the cart, favourites and
order items; /:id for the product screen, which today downloads the
whole catalogue to find one row; /facets for the filter sheet's options,
which the apps currently derive by scanning the catalogue in the
browser.
Deliberately not cached: CatalogCacheService exists because the unpaged
query is expensive and every screen wanted it. A paged, filtered query
is neither — bounded by limit and index-served — and caching it would
need a key per filter combination, i.e. unbounded memory for cheap
queries plus a second place for stale rows to hide.
D27's two storefront rules (active, has an image) live in one shared
where clause, so the list, its count, the single-product route, by-ids
and the facets can never disagree about what is visible. limit and ids
are both capped at 100; an uncapped limit is just the old "give me
everything" query spelled differently.
17 new cases, 329/329 overall. All 17 passed on the first run, which is
not evidence by itself, so one filter was deliberately broken to confirm
the suite catches it (1 failed/16 passed) and then restored.
Not measured on real data — the database container was not running, so
16k-row behaviour is reasoned from the query plan, not timed. Worth one
EXPLAIN ANALYZE on a full dataset before step 4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
bfbac92df5
commit
cbb93dec5d
@@ -1176,3 +1176,85 @@ guards, and `grep back-to=` across both apps. api suite untouched at 308/308.
|
||||
worth one pass by hand.
|
||||
|
||||
_No data was deleted in this entry._
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-12 — Catalogue paging, steps 1–2: indexes and the server routes
|
||||
|
||||
- **What:** `Product` and its two child tables get the indexes the storefront
|
||||
query actually needs, and `GET /catalog/products` learns paging, filtering,
|
||||
sorting, a by-ids mode, a single-product route and a facets route.
|
||||
- **Why:** owner — «دیتاها در اپ واقعی تا 16 هزار رکورد میرود و گرفتن همه یک جا
|
||||
اشتباه است، حتما باید پیج نیشن داشته باشد». The route had **no** paging
|
||||
parameters at all and returned every active product with its variants, images,
|
||||
category, brand and collections in one response; the apps' `PAGE_SIZE = 20`
|
||||
was only a render window over that full array, so scrolling made zero
|
||||
requests.
|
||||
- **Reference:** the reference DID page this server-side; the port moved it to
|
||||
the client. Logged as STUB-03, so this is a return to reference behaviour.
|
||||
- **Target:** `apps/api/prisma/schema.prisma`,
|
||||
`prisma/migrations/20261012120000_catalog_paging_indexes/`,
|
||||
`modules/catalog/catalog.controller.ts`, `catalog.service.ts`,
|
||||
`dto/list-products.dto.ts`, `catalog-paging.spec.ts`.
|
||||
- **Matrix row:** STUB-03 in `ISSUES.md`; plan in `docs/CATALOG-PAGINATION-PLAN.md`.
|
||||
- **Commit:** see below.
|
||||
|
||||
**The indexes were not optional.** `Product` carried **no index whatsoever** — not
|
||||
`isActive`, not `categoryId`, not `brandId`, not `createdAt` — and
|
||||
`ProductVariant` had none on `productId`, so each per-product include scanned
|
||||
its whole table. Adding `take`/`skip` on top of that would have paged the
|
||||
response without making the query any faster, which is the failure mode this
|
||||
change exists to avoid. `Product_isActive_createdAt_idx` is composite on purpose:
|
||||
it serves the filter, the default ordering and the OFFSET walk from one index.
|
||||
|
||||
`CREATE INDEX` and not `CONCURRENTLY`: the deploy image's own migration runner
|
||||
wraps each file in a transaction, which `CONCURRENTLY` cannot join (D15). At 16k
|
||||
rows the write lock is a few hundred milliseconds. Every statement is
|
||||
`IF NOT EXISTS`, because these run under both migration paths.
|
||||
|
||||
**Paging is opt-in, and that is the whole backward-compatibility story.** With no
|
||||
query parameters the route returns the same plain array from the same cache it
|
||||
always did — the admin panel, the smoke scripts and installed app builds are
|
||||
untouched. Pass `page` or `limit` (or any filter) and the answer becomes
|
||||
`{ rows, total, page, limit }`.
|
||||
|
||||
**The filters had to move server-side, not just the paging.** The apps filtered
|
||||
title, category, brand and gram range in the browser over the full array. Paging
|
||||
the response while leaving that in place would have made search **quietly
|
||||
wrong** — searching only the first page and reporting "محصولی پیدا نشد" for
|
||||
products that exist. That is worse than the slowness, so `storefrontWhere`
|
||||
builds every filter into the Prisma query, and `count` uses the same `where` so
|
||||
`total` describes the filtered set the app is scrolling.
|
||||
|
||||
**Three routes exist so the apps can stop fetching everything to find one thing**
|
||||
(step 3 wires them up): `?ids=a,b,c` for the cart, favourites and order items;
|
||||
`/:id` for the product screen, which currently downloads the entire catalogue to
|
||||
`find()` one row; and `/facets` for the filter sheet's options, which the apps
|
||||
derive today by scanning the whole catalogue in the browser.
|
||||
|
||||
**Deliberately NOT cached.** `CatalogCacheService` holds one built response
|
||||
because the unpaged query is expensive and every screen asked for it. A paged,
|
||||
filtered query is neither — it is bounded by `limit` and index-served. Caching it
|
||||
would need a key per filter combination: unbounded memory for queries that are
|
||||
already cheap, and a second place for stale rows to hide. Noted in the code so
|
||||
it is not "optimised" later.
|
||||
|
||||
**Guards kept everywhere.** D27's two storefront rules (active, and has at least
|
||||
one image) are in one shared `where`, so a paged list, its count, the single
|
||||
product route, the by-ids route and the facets can never disagree about what is
|
||||
visible. Asking for a hidden product by id still 404s. `limit` is capped at 100
|
||||
and `ids` at 100 — an uncapped `limit` would just be the old "give me
|
||||
everything" query spelled differently.
|
||||
|
||||
**Verification.** 17 new cases (329/329 overall, 29 suites), covering the
|
||||
compatibility shape, skip/take translation, that `total` counts the filtered set,
|
||||
each filter reaching the DB, the gram range matching ANY variant as the app did,
|
||||
by-ids ignoring paging, and a hidden product 404ing. All 17 passed on the first
|
||||
run, which is not evidence on its own — so one filter was deliberately broken to
|
||||
confirm the suite fails (it did, 1 failed/16 passed) and then restored.
|
||||
|
||||
⚠️ Measured on **no real data**: the database container was not running, so the
|
||||
16k-row behaviour is reasoned from the query plan and the new indexes, not timed.
|
||||
Worth one `EXPLAIN ANALYZE` on a full dataset before step 4.
|
||||
|
||||
_No data was deleted in this entry._
|
||||
|
||||
Reference in New Issue
Block a user