From cbb93dec5d02204399a7bbfb8d1c9b67470c158d Mon Sep 17 00:00:00 2001 From: "s.esmaeili" Date: Mon, 12 Oct 2026 15:01:32 +0330 Subject: [PATCH] feat(catalog): page and filter the storefront server-side (steps 1-2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The real catalogue reaches ~16k products and GET /catalog/products had no paging parameters at all: it returned every active product with its variants, images, category, brand and collections in one response. The apps' PAGE_SIZE = 20 was only a render window over that full array, so scrolling made zero requests. The reference paged this server-side; the port had moved it to the client (STUB-03), so this is a return to reference behaviour. Step 1 — indexes, which were not optional. Product carried NO index whatsoever (not isActive, categoryId, brandId or createdAt) and ProductVariant had none on productId, so every per-product include scanned its whole table. Adding take/skip on top of that would have paged the response without making the query faster, which is the exact failure this change exists to avoid. Product_isActive_createdAt_idx is composite so one index serves the filter, the default ordering and the OFFSET walk. CREATE INDEX, not CONCURRENTLY: the deploy image's migration runner wraps each file in a transaction, which CONCURRENTLY cannot join (D15). Every statement is IF NOT EXISTS because both migration paths run it. Step 2 — the routes. Paging is opt-in, which is the whole backward-compatibility story: with no query parameters the route returns the same plain array from the same cache as before, so the admin panel, the smoke scripts and installed app builds are untouched. Pass page, limit or any filter and it becomes { rows, total, page, limit }. The filters had to move too, not just the paging. The apps filtered title, category, brand and gram range in the browser over the full array; paging the response while leaving that in place would have made search quietly WRONG — searching only the first page and reporting "محصولی پیدا نشد" for products that exist. So every filter is built into the Prisma query, and count uses the same where clause, so `total` describes the set the app is actually scrolling. Three routes added so the apps can stop fetching everything to find one thing (wired up in step 3): ?ids=a,b,c for the cart, favourites and order items; /:id for the product screen, which today downloads the whole catalogue to find one row; /facets for the filter sheet's options, which the apps currently derive by scanning the catalogue in the browser. Deliberately not cached: CatalogCacheService exists because the unpaged query is expensive and every screen wanted it. A paged, filtered query is neither — bounded by limit and index-served — and caching it would need a key per filter combination, i.e. unbounded memory for cheap queries plus a second place for stale rows to hide. D27's two storefront rules (active, has an image) live in one shared where clause, so the list, its count, the single-product route, by-ids and the facets can never disagree about what is visible. limit and ids are both capped at 100; an uncapped limit is just the old "give me everything" query spelled differently. 17 new cases, 329/329 overall. All 17 passed on the first run, which is not evidence by itself, so one filter was deliberately broken to confirm the suite catches it (1 failed/16 passed) and then restored. Not measured on real data — the database container was not running, so 16k-row behaviour is reasoned from the query plan, not timed. Worth one EXPLAIN ANALYZE on a full dataset before step 4. Co-Authored-By: Claude Opus 5 --- .../migration.sql | 30 +++ apps/api/prisma/schema.prisma | 21 ++ .../modules/catalog/catalog-paging.spec.ts | 212 ++++++++++++++++++ .../src/modules/catalog/catalog.controller.ts | 50 ++++- .../src/modules/catalog/catalog.service.ts | 184 ++++++++++++++- .../modules/catalog/dto/list-products.dto.ts | 102 +++++++++ docs/CATALOG-PAGINATION-PLAN.md | 141 ++++++++++++ docs/parity/CHANGELOG.md | 82 +++++++ 8 files changed, 813 insertions(+), 9 deletions(-) create mode 100644 apps/api/prisma/migrations/20261012120000_catalog_paging_indexes/migration.sql create mode 100644 apps/api/src/modules/catalog/catalog-paging.spec.ts create mode 100644 apps/api/src/modules/catalog/dto/list-products.dto.ts create mode 100644 docs/CATALOG-PAGINATION-PLAN.md diff --git a/apps/api/prisma/migrations/20261012120000_catalog_paging_indexes/migration.sql b/apps/api/prisma/migrations/20261012120000_catalog_paging_indexes/migration.sql new file mode 100644 index 0000000..7df7624 --- /dev/null +++ b/apps/api/prisma/migrations/20261012120000_catalog_paging_indexes/migration.sql @@ -0,0 +1,30 @@ +-- Catalogue paging indexes. +-- +-- Owner, 2026-10-12: «دیتاها در اپ واقعی تا 16 هزار رکورد میرود و گرفتن همه یک +-- جا اشتباه است، حتما باید پیج نیشن داشته باشد.» Step 1 of +-- docs/CATALOG-PAGINATION-PLAN.md: without these, server-side paging would walk +-- the same full table scans the unpaged query did. +-- +-- `Product` carried NO index whatsoever, so every GET /catalog/products was a +-- sequential scan, and so was each per-product include. +-- +-- IF NOT EXISTS throughout: these run under BOTH migration paths (D15) — the +-- Prisma CLI in development and the app's own runner in the deploy image. +-- +-- CREATE INDEX (not CONCURRENTLY): the app's runner wraps each migration in a +-- transaction, which CONCURRENTLY cannot join. On a 16k-row table this locks +-- writes for a few hundred milliseconds, which is acceptable; a table large +-- enough to matter would need the index built by hand outside a migration. + +-- Matches the storefront list exactly: filter isActive, order by createdAt. +CREATE INDEX IF NOT EXISTS "Product_isActive_createdAt_idx" ON "Product" ("isActive", "createdAt"); + +-- The category and brand filters of the search screen. +CREATE INDEX IF NOT EXISTS "Product_categoryId_idx" ON "Product" ("categoryId"); +CREATE INDEX IF NOT EXISTS "Product_brandId_idx" ON "Product" ("brandId"); + +-- `include: { variants: { where: { isActive: true } } }`, once per product. +CREATE INDEX IF NOT EXISTS "ProductVariant_productId_isActive_idx" ON "ProductVariant" ("productId", "isActive"); + +-- `where: { images: { some: {} } }` plus the gallery ordered by sortOrder. +CREATE INDEX IF NOT EXISTS "ProductImage_productId_sortOrder_idx" ON "ProductImage" ("productId", "sortOrder"); diff --git a/apps/api/prisma/schema.prisma b/apps/api/prisma/schema.prisma index 69a07f0..559b91f 100644 --- a/apps/api/prisma/schema.prisma +++ b/apps/api/prisma/schema.prisma @@ -667,6 +667,18 @@ model Product { promotions Promotion[] allocationItems BranchAllocationItem[] favoritedBy CustomerFavorite[] + + // Storefront paging (owner: the real catalogue reaches ~16k products, so + // GET /catalog/products must page rather than return everything). This model + // had NO index at all, so every storefront query was a full scan and paging + // alone would not have been any faster. + // + // The composite matches the list's own shape: filter on isActive, order by + // createdAt — Postgres can satisfy both from this one index, including the + // OFFSET walk. The other two serve the category and brand filters. + @@index([isActive, createdAt]) + @@index([categoryId]) + @@index([brandId]) } model ProductImage { @@ -684,6 +696,11 @@ model ProductImage { /// variants (one photo, several weights), which is why this is a plain /// one-to-many and not a join table. variants ProductVariant[] + + // The storefront list filters on `images: { some: {} }` and includes the + // gallery ordered by sortOrder, both per product. Unindexed, each one scanned + // the whole table. + @@index([productId, sortOrder]) } enum ProductImageRole { @@ -721,6 +738,10 @@ model ProductVariant { orderItems OrderItem[] @@index([imageId]) + // Every storefront product carries its variants, so this include ran once per + // product with no index to use. The composite covers the `isActive: true` + // filter the list applies to them. + @@index([productId, isActive]) } // ───────────────────────────────────────────────────────────────────────── diff --git a/apps/api/src/modules/catalog/catalog-paging.spec.ts b/apps/api/src/modules/catalog/catalog-paging.spec.ts new file mode 100644 index 0000000..c6c00cc --- /dev/null +++ b/apps/api/src/modules/catalog/catalog-paging.spec.ts @@ -0,0 +1,212 @@ +import { NotFoundException } from '@nestjs/common'; +import { CatalogService } from './catalog.service'; +import { DEFAULT_PAGE_SIZE } from './dto/list-products.dto'; + +/** + * Server-side catalogue paging and filtering. + * + * Owner, 2026-10-12: «دیتاها در اپ واقعی تا 16 هزار رکورد میرود و گرفتن همه یک + * جا اشتباه است، حتما باید پیج نیشن داشته باشد.» + * + * Two things these cases exist to protect: + * + * 1. The bare `GET /catalog/products` must keep returning the whole array from + * the cache. The admin panel, the smoke scripts and already-installed app + * builds all depend on that shape, and paging is opt-in precisely so none + * of them break. + * 2. A filter must reach the DATABASE. Paging the response while leaving the + * filter in the client would make search quietly wrong — it would only ever + * search the first page — which is worse than the slowness being fixed. + */ +describe('CatalogService — paging and filtering', () => { + function setup(rowCount = 3) { + const rows = Array.from({ length: rowCount }, (_, i) => ({ + id: `p${i}`, + title: `محصول ${i}`, + isActive: true, + accountingStockWeight: null, + accountingStockCount: null, + accountingStockAt: null, + variants: [{ id: `v${i}`, isActive: true, isPreOrder: false, qty: 5 }], + images: [{ id: `i${i}`, sortOrder: 0 }], + })); + + const findMany = jest.fn().mockResolvedValue(rows); + const count = jest.fn().mockResolvedValue(42); + const prisma = { + product: { findMany, count }, + category: { findMany: jest.fn().mockResolvedValue([{ id: 'c1', title: 'انگشتر' }]) }, + brand: { findMany: jest.fn().mockResolvedValue([{ id: 'b1', title: 'برند' }]) }, + collection: { findMany: jest.fn().mockResolvedValue([{ id: 'col1', title: 'مجموعه' }]) }, + productVariant: { + aggregate: jest.fn().mockResolvedValue({ + _min: { rashWeightGrams: { toString: () => '1.5' } }, + _max: { rashWeightGrams: { toString: () => '12' } }, + }), + }, + }; + // Nothing is out of stock in the accounting software for these cases. + const accounting = { stockBlocker: jest.fn().mockResolvedValue(() => false) }; + const cache = { get: jest.fn((build: () => Promise) => build()) }; + const service = new CatalogService(prisma as never, accounting as never, cache as never); + return { service, findMany, count, prisma, cache }; + } + + /** The `where` of the last product.findMany call. */ + const lastWhere = (findMany: jest.Mock) => findMany.mock.calls.at(-1)?.[0]?.where; + const lastArgs = (findMany: jest.Mock) => findMany.mock.calls.at(-1)?.[0]; + + describe('backward compatibility', () => { + it('returns a plain array through the cache when nothing is asked for', async () => { + const { service, cache, findMany } = setup(); + const out = await service.listProducts(); + expect(Array.isArray(out)).toBe(true); + expect(cache.get).toHaveBeenCalledTimes(1); + // The original query: storefront rule only, no skip/take/orderBy. + const args = lastArgs(findMany); + expect(args.skip).toBeUndefined(); + expect(args.take).toBeUndefined(); + expect(args.where).toEqual({ isActive: true, images: { some: {} } }); + }); + + it('still applies the two storefront rules on a paged query', async () => { + // Inactive products and products with no photo must never appear, however + // the caller asks — D27. + const { service, findMany } = setup(); + await service.findProducts({ page: 1, limit: 10 }); + expect(lastWhere(findMany)).toMatchObject({ isActive: true, images: { some: {} } }); + }); + }); + + describe('paging', () => { + it('translates page/limit into skip/take', async () => { + const { service, findMany } = setup(); + await service.findProducts({ page: 3, limit: 20 }); + expect(lastArgs(findMany)).toMatchObject({ skip: 40, take: 20 }); + }); + + it('defaults the page size when only a page is given', async () => { + const { service, findMany } = setup(); + await service.findProducts({ page: 2 }); + expect(lastArgs(findMany)).toMatchObject({ skip: DEFAULT_PAGE_SIZE, take: DEFAULT_PAGE_SIZE }); + }); + + it('reports the total of the FILTERED set, not of the page', async () => { + // The app shows "N محصول" and stops scrolling by comparing against this, + // so counting anything else would both lie and break the end of the list. + const { service, count } = setup(); + const out = await service.findProducts({ page: 1, limit: 20, search: 'انگشتر' }); + expect(out.total).toBe(42); + expect(count).toHaveBeenCalledWith({ + where: expect.objectContaining({ title: { contains: 'انگشتر', mode: 'insensitive' } }), + }); + }); + + it('filters without paging when no page is given', async () => { + const { service, findMany } = setup(); + const out = await service.findProducts({ search: 'x' }); + expect(lastArgs(findMany).skip).toBeUndefined(); + expect(lastArgs(findMany).take).toBeUndefined(); + // `limit` echoes the total so a caller cannot mistake it for a page size. + expect(out.limit).toBe(out.total); + }); + + it('orders newest first by default, and honours the other sorts', async () => { + const { service, findMany } = setup(); + await service.findProducts({ page: 1 }); + expect(lastArgs(findMany).orderBy).toEqual({ createdAt: 'desc' }); + await service.findProducts({ page: 1, sort: 'title' }); + expect(lastArgs(findMany).orderBy).toEqual({ title: 'asc' }); + await service.findProducts({ page: 1, sort: 'oldest' }); + expect(lastArgs(findMany).orderBy).toEqual({ createdAt: 'asc' }); + }); + }); + + describe('filters reach the database', () => { + it('passes category, brand and collection through', async () => { + const { service, findMany } = setup(); + await service.findProducts({ page: 1, categoryId: 'c1', brandId: 'b1', collectionId: 'col1' }); + expect(lastWhere(findMany)).toMatchObject({ + categoryId: 'c1', + brandId: 'b1', + collections: { some: { collectionId: 'col1' } }, + }); + }); + + it('matches a gram range against the variants, as the app did', async () => { + // A product qualifies when ANY of its variants is in range + // (SearchView.vue:189-195), not when all of them are. + const { service, findMany } = setup(); + await service.findProducts({ page: 1, minGram: 2, maxGram: 8 }); + expect(lastWhere(findMany).variants).toEqual({ + some: { isActive: true, rashWeightGrams: { gte: 2, lte: 8 } }, + }); + }); + + it('accepts an open-ended gram range', async () => { + const { service, findMany } = setup(); + await service.findProducts({ page: 1, minGram: 5 }); + expect(lastWhere(findMany).variants).toEqual({ + some: { isActive: true, rashWeightGrams: { gte: 5 } }, + }); + }); + + it('ignores a blank search instead of searching for nothing', async () => { + const { service, findMany } = setup(); + await service.findProducts({ page: 1, search: ' ' }); + expect(lastWhere(findMany).title).toBeUndefined(); + }); + }); + + describe('by ids — the cart, favourites and order items', () => { + it('fetches only the named products and skips paging', async () => { + const { service, findMany } = setup(2); + const out = await service.findProducts({ ids: ['p0', 'p1'], page: 5, limit: 1 }); + expect(lastWhere(findMany)).toMatchObject({ id: { in: ['p0', 'p1'] } }); + expect(lastArgs(findMany).skip).toBeUndefined(); + expect(out.total).toBe(2); + }); + + it('still hides an inactive or photo-less product asked for by id', async () => { + const { service, findMany } = setup(1); + await service.findProducts({ ids: ['p0'] }); + expect(lastWhere(findMany)).toMatchObject({ isActive: true, images: { some: {} } }); + }); + }); + + describe('one product by id', () => { + it('returns it', async () => { + const { service } = setup(1); + await expect(service.findProductById('p0')).resolves.toMatchObject({ id: 'p0' }); + }); + + it('404s for a product that is not on the storefront', async () => { + const { service, findMany } = setup(0); + findMany.mockResolvedValue([]); + await expect(service.findProductById('nope')).rejects.toThrow(NotFoundException); + }); + }); + + describe('facets', () => { + it('returns the filter options as small aggregates', async () => { + // These used to be derived by scanning the whole catalogue in the browser, + // which only works while the whole catalogue is in memory. + const { service } = setup(); + const out = await service.facets(); + expect(out.categories).toEqual([{ id: 'c1', title: 'انگشتر' }]); + expect(out.brands).toEqual([{ id: 'b1', title: 'برند' }]); + expect(out.collections).toEqual([{ id: 'col1', title: 'مجموعه' }]); + // Strings, like every other weight this API returns — Decimal, not float. + expect(out.gramRange).toEqual({ min: '1.5', max: '12' }); + }); + + it('reports a zero range for an empty catalogue', async () => { + const { service, prisma } = setup(); + prisma.productVariant.aggregate.mockResolvedValue({ + _min: { rashWeightGrams: null }, + _max: { rashWeightGrams: null }, + }); + await expect(service.facets()).resolves.toMatchObject({ gramRange: { min: '0', max: '0' } }); + }); + }); +}); diff --git a/apps/api/src/modules/catalog/catalog.controller.ts b/apps/api/src/modules/catalog/catalog.controller.ts index 31cab97..725ceff 100644 --- a/apps/api/src/modules/catalog/catalog.controller.ts +++ b/apps/api/src/modules/catalog/catalog.controller.ts @@ -1,14 +1,58 @@ -import { Controller, Get } from '@nestjs/common'; +import { Controller, Get, Param, Query } from '@nestjs/common'; import { CatalogService } from './catalog.service'; import { Public } from '../auth/guards'; +import { ListProductsDto } from './dto/list-products.dto'; @Public() // pre-existing unauthenticated endpoint; see Phase 1 notes @Controller('catalog/products') export class CatalogController { constructor(private readonly catalog: CatalogService) {} + /** + * The storefront list. + * + * Paging and filtering are OPT-IN, so this stays backward compatible: with no + * query at all the caller gets the plain array it always got (the admin panel, + * the smoke scripts and installed app builds all rely on that). Pass `page` or + * `limit` and the answer becomes `{ rows, total, page, limit }`. + * + * Owner, 2026-10-12: the real catalogue reaches ~16k products, so a client + * must be able to ask for one page — and the filters have to be applied here, + * because a client cannot filter rows it was never sent. + */ @Get() - list() { - return this.catalog.listProducts(); + list(@Query() query: ListProductsDto) { + const asksForMore = + query.page !== undefined || + query.limit !== undefined || + query.search !== undefined || + query.categoryId !== undefined || + query.brandId !== undefined || + query.collectionId !== undefined || + query.minGram !== undefined || + query.maxGram !== undefined || + query.sort !== undefined || + query.ids !== undefined; + + // The bare call keeps its original shape — a plain array, from the cache. + return asksForMore ? this.catalog.findProducts(query) : this.catalog.listProducts(); + } + + /** + * The filter options for the search sheet. + * + * Declared BEFORE `:id`: Nest matches in declaration order, so a literal + * segment that could also read as a parameter has to come first or it is + * never reached. + */ + @Get('facets') + facets() { + return this.catalog.facets(); + } + + /** One product, for its own screen. */ + @Get(':id') + byId(@Param('id') id: string) { + return this.catalog.findProductById(id); } } diff --git a/apps/api/src/modules/catalog/catalog.service.ts b/apps/api/src/modules/catalog/catalog.service.ts index 03565ce..d836097 100644 --- a/apps/api/src/modules/catalog/catalog.service.ts +++ b/apps/api/src/modules/catalog/catalog.service.ts @@ -1,7 +1,9 @@ import { BadRequestException, Injectable, NotFoundException } from '@nestjs/common'; +import { Prisma } from '@prisma/client'; import { PrismaService } from '../../prisma/prisma.service'; import { AccountingHooksService } from '../accounting-sync/accounting-hooks.service'; import { CatalogCacheService } from './catalog-cache.service'; +import { DEFAULT_PAGE_SIZE, ListProductsDto, type ProductSort } from './dto/list-products.dto'; @Injectable() export class CatalogService { @@ -11,17 +13,187 @@ export class CatalogService { private readonly catalogCache: CatalogCacheService, ) {} - /** Served from CatalogCacheService; see there for what invalidates it. */ + /** + * The whole active catalogue, unpaged. + * + * Kept exactly as it was for every existing caller (the admin panel, the smoke + * scripts, installed app builds). New callers should use `findProducts` with a + * page — the real catalogue reaches ~16k products, and this response carries + * all of them with their variants, images and collections. + * + * Served from CatalogCacheService; see there for what invalidates it. + */ listProducts() { return this.catalogCache.get(() => this.buildProductList()); } - private async buildProductList() { + /** + * The storefront list, paged and filtered server-side. + * + * Owner, 2026-10-12: the catalogue reaches ~16k products, so returning it + * whole is wrong. The filters live here rather than in the apps because a + * client cannot filter data it has not been sent — paging the response while + * leaving the filter on the client would silently search only the first page. + * + * Returns `{ rows, total, page, limit }`. With no `page`/`limit` the caller + * gets everything that matches, so the filters are usable on their own. + * + * NOT cached, deliberately. CatalogCacheService holds one built response and + * exists because the unpaged query is expensive and every screen asked for + * it. A paged, filtered query is neither: it is bounded by `limit` and served + * by Product_isActive_createdAt_idx. Caching it would mean a key per filter + * combination — unbounded memory for queries that are already cheap, and a + * second place for stale rows to hide. If this ever needs a cache it should + * be a shared one (Redis), which is also what the single-instance note on + * CatalogCacheService points at. + */ + async findProducts(query: ListProductsDto) { + // Specific ids: the cart, favourites and order items each need a handful of + // known products. Paging and the filters do not apply. + if (query.ids?.length) { + const rows = await this.buildProductList({ ids: query.ids }); + return { rows, total: rows.length, page: 1, limit: rows.length }; + } + + const paged = query.page !== undefined || query.limit !== undefined; + const limit = query.limit ?? DEFAULT_PAGE_SIZE; + const page = query.page ?? 1; + + const where = this.storefrontWhere(query); + // Counted with the same `where`, so `total` describes the filtered set and + // the app's "N محصول" and end-of-list detection stay truthful. + const [total, rows] = await Promise.all([ + this.prisma.product.count({ where }), + this.buildProductList({ + where, + orderBy: this.orderFor(query.sort), + ...(paged ? { skip: (page - 1) * limit, take: limit } : {}), + }), + ]); + + return { rows, total, page: paged ? page : 1, limit: paged ? limit : total }; + } + + /** One product for its own screen, instead of fetching the catalogue to find it. */ + async findProductById(id: string) { + const [product] = await this.buildProductList({ where: { ...this.storefrontWhere({}), id } }); + if (!product) throw new NotFoundException('محصول یافت نشد'); + return product; + } + + /** + * The filter options the search sheet offers. + * + * The apps derived these by scanning the whole catalogue in the browser + * (SearchView.vue:167-181), which is only possible while the whole catalogue + * is in memory. Each is a small aggregate here. + */ + async facets() { + const where = this.storefrontWhere({}); + const [categories, brands, collections, gram] = await Promise.all([ + this.prisma.category.findMany({ + where: { products: { some: where } }, + select: { id: true, title: true }, + orderBy: { title: 'asc' }, + }), + this.prisma.brand.findMany({ + where: { products: { some: where } }, + select: { id: true, title: true }, + orderBy: { title: 'asc' }, + }), + this.prisma.collection.findMany({ + where: { products: { some: { product: where } } }, + select: { id: true, title: true }, + orderBy: { title: 'asc' }, + }), + this.prisma.productVariant.aggregate({ + where: { isActive: true, product: where }, + _min: { rashWeightGrams: true }, + _max: { rashWeightGrams: true }, + }), + ]); + + return { + categories, + brands, + collections, + // Strings, like every other weight this API returns (Decimal, not float). + gramRange: { + min: gram._min.rashWeightGrams?.toString() ?? '0', + max: gram._max.rashWeightGrams?.toString() ?? '0', + }, + }; + } + + /** + * The two rules that decide whether a product is on the storefront at all, + * plus the caller's filters. Shared so a filtered list, its count, the single + * product route and the facets can never disagree about what is visible. + * + * D27 — a product with no image is never shown in the app. Products also + * arrive from the accounting software, which carries no pictures; they appear + * once an admin photographs them. + */ + private storefrontWhere(query: ListProductsDto): Prisma.ProductWhereInput { + const where: Prisma.ProductWhereInput = { isActive: true, images: { some: {} } }; + + if (query.search?.trim()) { + // `contains`, as the apps' own title filter did. Case-insensitive is a + // no-op for Persian but matches how every other search in this API reads. + where.title = { contains: query.search.trim(), mode: 'insensitive' }; + } + if (query.categoryId) where.categoryId = query.categoryId; + if (query.brandId) where.brandId = query.brandId; + if (query.collectionId) where.collections = { some: { collectionId: query.collectionId } }; + + // A product is in range when at least ONE of its variants is, which is what + // the app's `limit_gram` filter meant (SearchView.vue:189-195). + if (query.minGram !== undefined || query.maxGram !== undefined) { + where.variants = { + some: { + isActive: true, + rashWeightGrams: { + ...(query.minGram !== undefined ? { gte: query.minGram } : {}), + ...(query.maxGram !== undefined ? { lte: query.maxGram } : {}), + }, + }, + }; + } + + return where; + } + + private orderFor(sort: ProductSort | undefined): Prisma.ProductOrderByWithRelationInput { + switch (sort) { + case 'oldest': + return { createdAt: 'asc' }; + case 'title': + return { title: 'asc' }; + default: + // Newest first, and the half of Product_isActive_createdAt_idx the list + // is indexed for. + return { createdAt: 'desc' }; + } + } + + private async buildProductList( + opts: { + where?: Prisma.ProductWhereInput; + orderBy?: Prisma.ProductOrderByWithRelationInput; + skip?: number; + take?: number; + ids?: string[]; + } = {}, + ) { const products = await this.prisma.product.findMany({ - // D27 — a product with no image is never shown in the app. Products now - // also arrive from the accounting software, which carries no pictures; - // they appear once an admin photographs them. - where: { isActive: true, images: { some: {} } }, + // Defaults to the unfiltered storefront rule, so the unpaged + // `listProducts()` above behaves exactly as before. + where: opts.ids + ? { ...this.storefrontWhere({}), id: { in: opts.ids } } + : (opts.where ?? { isActive: true, images: { some: {} } }), + ...(opts.orderBy ? { orderBy: opts.orderBy } : {}), + ...(opts.skip !== undefined ? { skip: opts.skip } : {}), + ...(opts.take !== undefined ? { take: opts.take } : {}), include: { // Only orderable variants reach the storefront. An inactive one used to // be listed and then rejected by getVariantOrThrow at checkout, so the diff --git a/apps/api/src/modules/catalog/dto/list-products.dto.ts b/apps/api/src/modules/catalog/dto/list-products.dto.ts new file mode 100644 index 0000000..91452ff --- /dev/null +++ b/apps/api/src/modules/catalog/dto/list-products.dto.ts @@ -0,0 +1,102 @@ +import { Transform, Type } from 'class-transformer'; +import { ArrayMaxSize, IsArray, IsIn, IsInt, IsNumber, IsOptional, IsString, Max, Min } from 'class-validator'; + +/** Page size when the caller asks for paging without naming one. */ +export const DEFAULT_PAGE_SIZE = 20; +/** + * Hard ceiling on `limit`. + * + * The real catalogue reaches ~16k products, so an unbounded `limit` would let + * any caller ask for all of it — the exact query this whole change exists to + * stop, just spelled differently. 100 is five of the apps' own pages. + */ +export const MAX_PAGE_SIZE = 100; +/** + * Ceiling on `ids`. The cart, favourites and an order's items are all small; + * anything larger is a caller that should be paging instead. + */ +export const MAX_IDS = 100; + +export type ProductSort = 'newest' | 'oldest' | 'title'; + +/** + * Query for `GET /catalog/products`. + * + * Paging is OPT-IN: with neither `page` nor `limit` the route answers exactly as + * it always did, with the whole catalogue, so the admin panel, the smoke scripts + * and already-installed app builds keep working (plan §3-ب). The filters are + * honoured either way. + */ +export class ListProductsDto { + /** 1-based. Sending it (or `limit`) is what turns paging on. */ + @IsOptional() + @Type(() => Number) + @IsInt() + @Min(1) + page?: number; + + @IsOptional() + @Type(() => Number) + @IsInt() + @Min(1) + @Max(MAX_PAGE_SIZE) + limit?: number; + + /** Free text over the product title. Matches the app's own client-side filter. */ + @IsOptional() + @IsString() + search?: string; + + @IsOptional() + @IsString() + categoryId?: string; + + @IsOptional() + @IsString() + brandId?: string; + + /** Collection ("مجموعه") the product belongs to — the collection screen. */ + @IsOptional() + @IsString() + collectionId?: string; + + /** + * Rash-weight range, matched against the product's VARIANTS: a product is in + * range when at least one of its variants is, which is what the app's own + * `limit_gram` filter did (SearchView.vue:189-195). + */ + @IsOptional() + @Type(() => Number) + @IsNumber() + @Min(0) + minGram?: number; + + @IsOptional() + @Type(() => Number) + @IsNumber() + @Min(0) + maxGram?: number; + + /** + * Specific ids, for the cart, favourites and order items — each needs a few + * known products, not a page. Ignores paging and the filters above. + * + * Comma-separated so this stays a plain cacheable GET (`?ids=a,b,c`) rather + * than needing a POST for a read. Capped, because the point of this parameter + * is a handful of products: a caller that wants more should page. + */ + @IsOptional() + @Transform(({ value }: { value: unknown }) => + typeof value === 'string' + ? value.split(',').map((s) => s.trim()).filter(Boolean).slice(0, MAX_IDS) + : value, + ) + @IsArray() + @ArrayMaxSize(MAX_IDS) + @IsString({ each: true }) + ids?: string[]; + + @IsOptional() + @IsIn(['newest', 'oldest', 'title']) + sort?: ProductSort; +} diff --git a/docs/CATALOG-PAGINATION-PLAN.md b/docs/CATALOG-PAGINATION-PLAN.md new file mode 100644 index 0000000..6d7c95e --- /dev/null +++ b/docs/CATALOG-PAGINATION-PLAN.md @@ -0,0 +1,141 @@ +# طرح — صفحه‌بندی واقعی کاتالوگ (۱۶ هزار رکورد) + +تاریخ: ۱۴۰۵/۰۷/۲۰ (2026-10-12) — **طرح؛ منتظر تأیید** + +> **مالک:** «دیتاها در اپ واقعی تا ۱۶ هزار رکورد می‌رود و گرفتن همه یک‌جا اشتباه +> است، حتماً باید پیجینیشن داشته باشد.» + +موافقم. با این عدد، وضعیت فعلی قابل دفاع نیست. این سند می‌گوید دقیقاً چه چیزی +شکسته است، چرا کار از «اضافه کردن `take`/`skip`» بزرگ‌تر است، و با چه ترتیبی +بی‌ریسک انجام می‌شود. + +--- + +## ۱. وضعیت امروز (بررسی‌شده در کد، نه حدس) + +| لایه | واقعیت | مدرک | +|---|---|---| +| مسیر API | `GET /catalog/products` **هیچ پارامتری نمی‌پذیرد** | `catalog.controller.ts:10-12` | +| کوئری | یک `findMany` بدون `take`/`skip`، همراه `variants`+`images`+`category`+`brand`+`collections` | `catalog.service.ts:19-42` | +| کش سرور | کل خروجی، ۳۰ ثانیه در حافظه، با هر نوشتن باطل | `catalog-cache.service.ts` | +| «صفحه‌بندی» اپ | `PAGE_SIZE = 20` فقط یک **پنجره رندر** روی آرایه‌ی کامل است: `slice(0, visibleCount)` | `SearchView.vue:204-206`، `NewestView.vue:55-57` | +| در هر اسکرول | **صفر درخواست**، ۲۰ کارت بیشتر رندر می‌شود | همان | + +**ایندکس‌ها — یافته‌ی مهم:** مدل `Product` **هیچ ایندکسی ندارد** (نه +`isActive`، نه `categoryId`، نه `brandId`، نه `createdAt`) و `ProductVariant` +ایندکسی روی `productId` ندارد. با ۱۶ هزار رکورد، صفحه‌بندی بدون ایندکس همان +کندی را می‌دهد؛ پس ایندکس **بخشی از این کار است**، نه یک بهبود جداگانه. + +## ۲. چرا کار از «اضافه کردن take/skip» بزرگ‌تر است + +هفت مصرف‌کننده روی فرض «کل کاتالوگ در حافظه است» نوشته شده‌اند. سه‌تا بحرانی‌اند +— هر سه کل لیست را می‌گیرند تا **یک چیز** پیدا کنند: + +| جا | الان چه می‌کند | با ۱۶ هزار رکورد | +|---|---|---| +| `useProductDetail` | کل کاتالوگ را می‌گیرد تا **یک محصول** را `find` کند | بازکردن هر محصول = دانلود کل کاتالوگ | +| `cartStore.init` | کل کاتالوگ برای همگام‌سازی **۲-۳ قلم سبد** | هر بار باز شدن اپ | +| `configStore.init` | کل کاتالوگ فقط برای **لیست دسته‌ها** | صفحه خانه | +| `SearchView` | متن/دسته/برند/بازه وزن را روی کلاینت فیلتر می‌کند | فیلتر روی داده‌ی ناقص = نتیجه غلط | +| `Filters.vue` | گزینه‌های فیلتر را از کل لیست مشتق می‌کند | گزینه‌ها ناقص می‌شوند | +| `FavoritesView` | علاقه‌مندی‌ها را با `find` در کل لیست پیدا می‌کند | علاقه‌مندیِ خارج از صفحه دیده نمی‌شود | +| `useOrderItemProduct` | عنوان محصولِ آیتم سفارش را از کل لیست می‌خواند | سفارش قدیمی بی‌عنوان می‌شود | + +**نتیجه:** اگر فقط `take`/`skip` به مسیر اضافه شود، جستجو و فیلتر **بی‌صدا غلط** +می‌شوند (روی ۲۰ رکورد اول فیلتر می‌کنند) و این بدتر از کندی است. پس هر +مصرف‌کننده باید مسیر مخصوص خودش را بگیرد. + +## ۳. طراحی پیشنهادی + +### (الف) چهار مسیر، به‌جای یک مسیر + +| مسیر | برای چه | چرا جدا | +|---|---|---| +| `GET /catalog/products?page=&limit=&search=&categoryId=&brandId=&minGram=&maxGram=&sort=` | لیست و جستجو | **فیلتر و صفحه‌بندی سمت سرور**؛ تنها راه درست بودن نتیجه | +| `GET /catalog/products/:id` | صفحه محصول | یک رکورد، نه ۱۶ هزار | +| `GET /catalog/facets` | گزینه‌های فیلتر (دسته‌ها، برندها، بازه وزن) | کوچک، `distinct`/`aggregate`؛ جایگزین مشتق‌کردن از کل لیست | +| `POST /catalog/products/by-ids` | سبد، علاقه‌مندی، آیتم‌های سفارش | چند شناسه مشخص در یک درخواست | + +شکل پاسخ لیست: `{ rows, total, page, limit }`. **عمداً** از `TableService` +استفاده نمی‌کنم: آن برای قرارداد XTable پنل ادمین است و شکل پاسخش +(`{rows, query:{...}}`) برای اپ بی‌مورد پیچیده است. + +### (ب) سازگاری عقب‌رو — مهم + +مسیر فعلی **نباید بشکند**: پنل ادمین، اسکریپت‌های smoke و نسخه‌های نصب‌شده‌ی اپ +از آن استفاده می‌کنند. + +- بدون `page`/`limit` ⇒ **رفتار امروز** (کل لیست). پس هیچ مصرف‌کننده‌ای نمی‌شکند. +- با `page`/`limit` ⇒ پاسخ صفحه‌بندی‌شده. +- در لاگ، درخواست بدون `page` روی کاتالوگ بزرگ **هشدار** می‌دهد تا بفهمیم چه + کسی هنوز قدیمی است. + +### (ج) کش سرور — بازطراحی + +کش فعلی **یک** خروجی را نگه می‌دارد (`value: {at, data}`). با پارامترها این +نادرست می‌شود: صفحهٔ ۲ جواب صفحهٔ ۱ را می‌گیرد. + +پیشنهاد: کلید کش = رشته‌ی پارامترهای نرمال‌شده، با سقف تعداد کلید (مثلاً ۵۰، +LRU) تا حافظه بی‌نهایت رشد نکند. باطل‌سازی موجود (middleware پریزما) **همه** را +با هم پاک می‌کند — که درست است، چون هر نوشتنی می‌تواند هر صفحه‌ای را عوض کند. + +⚠️ سقف کلید لازم است: بدون آن، هر ترکیب فیلترِ کاربران یک ردیف کش می‌سازد و +حافظه‌ی سرور به‌تدریج پر می‌شود. + +### (د) ایندکس‌ها (مهاجرت) + +``` +Product: @@index([isActive, createdAt]) ← لیست و مرتب‌سازی پیش‌فرض + @@index([categoryId]) + @@index([brandId]) +ProductVariant: @@index([productId]) ← الان نیست؛ هر include یک اسکن +ProductImage: @@index([productId]) +``` + +جستجوی متنی: `contains` فارسی روی ۱۶ هزار ردیف با ایندکس معمولی سریع نمی‌شود. +پیشنهاد **فعلی**: همان `contains` با ایندکس بالا (برای ۱۶ هزار قابل قبول است). +اگر کند بود، مرحله‌ی بعد `pg_trgm` است — ولی **اول اندازه‌گیری، بعد اضافه کردن**. + +### (ه) سمت اپ + +| کار | تغییر | +|---|---| +| `catalogStore` | از «یک آرایه‌ی کامل» به «صفحات + `total`»؛ `load()` صفحه می‌گیرد، `loadMore()` صفحه بعد | +| `SearchView` / `NewestView` | `IntersectionObserver` به‌جای زیاد کردن `visibleCount`، **درخواست صفحه بعد** می‌زند؛ فیلتر/جستجو با debounce به سرور می‌رود و از صفحه ۱ شروع می‌کند | +| `Filters.vue` | گزینه‌ها از `/catalog/facets` | +| `useProductDetail` | `GET /catalog/products/:id` | +| `cartStore.init` | `by-ids` با شناسه‌های سبد | +| `FavoritesView` | `by-ids` با شناسه‌های علاقه‌مندی | +| `useOrderItemProduct` | `by-ids`، یا بهتر: عنوان از خود `OrderItem` snapshot (اگر دارد) | +| `configStore` | دسته‌ها از `/catalog/facets` | + +**اندازه صفحه:** همان **۲۰** که الان پنجره‌ی رندر است، تا تجربه عوض نشود. + +## ۴. ترتیب اجرا (هر مرحله قابل تست و بی‌شکست) + +| مرحله | کار | چرا این ترتیب | +|---|---|---| +| ۱ | مهاجرت ایندکس‌ها | بدون این، صفحه‌بندی هم کند است. مستقل و بی‌ریسک. | +| ۲ | چهار مسیر سرور + کش کلیدی + تست | سازگار عقب‌رو؛ اپ‌ها هنوز دست‌نخورده کار می‌کنند | +| ۳ | `useProductDetail`، `cartStore`، `FavoritesView`، `configStore` → مسیرهای نقطه‌ای | بزرگ‌ترین برد با کمترین ریسک؛ رفتار دیداری عوض نمی‌شود | +| ۴ | `SearchView` / `NewestView` → صفحه‌بندی واقعی سرور | پرریسک‌ترین (فیلتر + اسکرول)؛ آخر، وقتی بقیه پایدار است | +| ۵ | حذف خوانش کل کاتالوگ و هشدار لاگ | فقط وقتی هیچ مصرف‌کننده‌ای نمانده | + +## ۵. دو چیز که باید بدانی + +**۱. مرجع صفحه‌بندی سرور داشت.** `ApiMobileController` برای لیست محصولات +صفحه‌بندی می‌فرستاد؛ پورت فعلی آن را به کلاینت منتقل کرده. پس این کار +**بازگشت به رفتار مرجع** است، نه قابلیت جدید — در ماتریس به‌عنوان STUB-03 ثبت +است. + +**۲. ریسک واقعی این کار، جستجو و فیلتر است.** امروز روی کلاینت فوری و دقیق +کار می‌کنند. منتقل‌شدن به سرور یعنی: debounce، حالت بارگذاری، و «نتیجه‌ای نیست» +که باید با صفحه‌ی خالی اشتباه نشود. مرحله ۴ را عمداً آخر گذاشتم و پیشنهادم این +است که وقتی رسیدیم، **یک بار با هم در مرورگر امتحانش کنیم** — چون این بخش +تنها جایی است که تست واحد کفایت نمی‌کند. + +--- + +**برای تصمیم تو:** مرحله ۱ و ۲ کاملاً بی‌ریسک‌اند و هیچ‌چیز را نمی‌شکنند. اگر +بگویی «برو»، از آن‌ها شروع می‌کنم و بعد از هر مرحله گزارش می‌دهم. اگر عجله داری، +مرحله ۳ همان‌جایی است که کاربر فوراً تفاوت را حس می‌کند (باز شدن صفحه محصول). diff --git a/docs/parity/CHANGELOG.md b/docs/parity/CHANGELOG.md index 6e1bda6..500bd54 100644 --- a/docs/parity/CHANGELOG.md +++ b/docs/parity/CHANGELOG.md @@ -1176,3 +1176,85 @@ guards, and `grep back-to=` across both apps. api suite untouched at 308/308. worth one pass by hand. _No data was deleted in this entry._ + +--- + +## 2026-10-12 — Catalogue paging, steps 1–2: indexes and the server routes + +- **What:** `Product` and its two child tables get the indexes the storefront + query actually needs, and `GET /catalog/products` learns paging, filtering, + sorting, a by-ids mode, a single-product route and a facets route. +- **Why:** owner — «دیتاها در اپ واقعی تا 16 هزار رکورد میرود و گرفتن همه یک جا + اشتباه است، حتما باید پیج نیشن داشته باشد». The route had **no** paging + parameters at all and returned every active product with its variants, images, + category, brand and collections in one response; the apps' `PAGE_SIZE = 20` + was only a render window over that full array, so scrolling made zero + requests. +- **Reference:** the reference DID page this server-side; the port moved it to + the client. Logged as STUB-03, so this is a return to reference behaviour. +- **Target:** `apps/api/prisma/schema.prisma`, + `prisma/migrations/20261012120000_catalog_paging_indexes/`, + `modules/catalog/catalog.controller.ts`, `catalog.service.ts`, + `dto/list-products.dto.ts`, `catalog-paging.spec.ts`. +- **Matrix row:** STUB-03 in `ISSUES.md`; plan in `docs/CATALOG-PAGINATION-PLAN.md`. +- **Commit:** see below. + +**The indexes were not optional.** `Product` carried **no index whatsoever** — not +`isActive`, not `categoryId`, not `brandId`, not `createdAt` — and +`ProductVariant` had none on `productId`, so each per-product include scanned +its whole table. Adding `take`/`skip` on top of that would have paged the +response without making the query any faster, which is the failure mode this +change exists to avoid. `Product_isActive_createdAt_idx` is composite on purpose: +it serves the filter, the default ordering and the OFFSET walk from one index. + +`CREATE INDEX` and not `CONCURRENTLY`: the deploy image's own migration runner +wraps each file in a transaction, which `CONCURRENTLY` cannot join (D15). At 16k +rows the write lock is a few hundred milliseconds. Every statement is +`IF NOT EXISTS`, because these run under both migration paths. + +**Paging is opt-in, and that is the whole backward-compatibility story.** With no +query parameters the route returns the same plain array from the same cache it +always did — the admin panel, the smoke scripts and installed app builds are +untouched. Pass `page` or `limit` (or any filter) and the answer becomes +`{ rows, total, page, limit }`. + +**The filters had to move server-side, not just the paging.** The apps filtered +title, category, brand and gram range in the browser over the full array. Paging +the response while leaving that in place would have made search **quietly +wrong** — searching only the first page and reporting "محصولی پیدا نشد" for +products that exist. That is worse than the slowness, so `storefrontWhere` +builds every filter into the Prisma query, and `count` uses the same `where` so +`total` describes the filtered set the app is scrolling. + +**Three routes exist so the apps can stop fetching everything to find one thing** +(step 3 wires them up): `?ids=a,b,c` for the cart, favourites and order items; +`/:id` for the product screen, which currently downloads the entire catalogue to +`find()` one row; and `/facets` for the filter sheet's options, which the apps +derive today by scanning the whole catalogue in the browser. + +**Deliberately NOT cached.** `CatalogCacheService` holds one built response +because the unpaged query is expensive and every screen asked for it. A paged, +filtered query is neither — it is bounded by `limit` and index-served. Caching it +would need a key per filter combination: unbounded memory for queries that are +already cheap, and a second place for stale rows to hide. Noted in the code so +it is not "optimised" later. + +**Guards kept everywhere.** D27's two storefront rules (active, and has at least +one image) are in one shared `where`, so a paged list, its count, the single +product route, the by-ids route and the facets can never disagree about what is +visible. Asking for a hidden product by id still 404s. `limit` is capped at 100 +and `ids` at 100 — an uncapped `limit` would just be the old "give me +everything" query spelled differently. + +**Verification.** 17 new cases (329/329 overall, 29 suites), covering the +compatibility shape, skip/take translation, that `total` counts the filtered set, +each filter reaching the DB, the gram range matching ANY variant as the app did, +by-ids ignoring paging, and a hidden product 404ing. All 17 passed on the first +run, which is not evidence on its own — so one filter was deliberately broken to +confirm the suite fails (it did, 1 failed/16 passed) and then restored. + +⚠️ Measured on **no real data**: the database container was not running, so the +16k-row behaviour is reasoned from the query plan and the new indexes, not timed. +Worth one `EXPLAIN ANALYZE` on a full dataset before step 4. + +_No data was deleted in this entry._