feat(replicas): auto-heal slave data drift and no-store public settings

Add sync-replica-data repair on deploy, a publisher-side replica watch
(60s + after tenant/settings writes), REFRESH PUBLICATION after migrate,
and Cache-Control no-store on program-settings and get_version.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-08-18 12:32:30 +03:30
co-authored by Cursor
parent 668bbcc1db
commit 16d562caa5
13 changed files with 456 additions and 11 deletions
+59 -5
View File
@@ -122,15 +122,69 @@ cp deploy/replicas.env.example deploy/replicas.env # once; add real DSNs
bash deploy/sync-replica-schema.sh --init # once, when schemas already match
```
For every release:
For every release on **publisher** (`/var/www/superadmin/api`):
```bash
npm run replicas:sync # DDL to each replica first, then migrate deploy here
npm run replicas:check # subscription state + columns missing on any replica
git pull
npm run replicas:sync # 1) DDL on replicas 2) migrate deploy 3) repair data drift if any
npx prisma generate
npm run build
pm2 restart superadmin-api
```
Keep migrations schema-only. Data backfills would run once per replica and then
again through replication, so apply those on the publisher by hand.
Check only:
```bash
npm run replicas:check # schema + subscription + row counts (settings, tenants, …)
```
Force data realign from publisher (rare — e.g. after fixing subscriber permissions):
```bash
npm run replicas:repair-data
```
`replicas:sync` runs `sync-replica-data.sh --repair-if-needed` automatically when
`settings` / `setting_templates` / `tenants` counts differ from publisher.
On the **publisher API** (`superadmin-api`), a replica watch also runs: ~8s after
boot, then every 60s, and immediately after tenant/settings writes. Drift is
healed without an operator command. Logical replication still streams live
INSERT/UPDATE/DELETE; the watch only fills **missed history** (WAL is not
rewound).
Each replica API user (`montime` by default) needs `SELECT` on all tables — the
repair script re-applies grants. `REPLICA_DSNS` must connect as a role that can
`TRUNCATE` + `ALTER SUBSCRIPTION` on the subscriber. The watch is idle unless
`deploy/replicas.env` exists on that host.
Keep migrations schema-only. After each migrate, subscribers get
`ALTER SUBSCRIPTION … REFRESH PUBLICATION WITH (copy_data = true)` so **new
tables** join replication (Postgres does not add them by itself).
## 9) Data changes **without** deploy (import settings, edit tenant, …)
**Normal path:** publisher writes → `superadmin_pub` WAL → slaves apply in
**seconds**. No `replicas:sync` for that.
**Why LSN-healthy slaves can still have empty `settings`:** logical replication
is a **stream**, not a mirror. `received_lsn = latest_end_lsn` means “caught up
with current WAL”, not “every historical row exists”. Rows inserted while apply
was broken, or while the table was **not yet in the subscription**, are never
replayed. Postgres will not go back.
That gap is healed **automatically on the publisher API** (watch every 60s +
right after import/edit). You do **not** run `replicas:repair-data` by hand
once this build is running on `/var/www/superadmin/api`.
| Action | What actually happens |
|--------|------------------------|
| INSERT/UPDATE/DELETE while subscription is ready | Live WAL — slaves update immediately |
| New table after `FOR ALL TABLES` publication | Not replicated until `REFRESH PUBLICATION` (deploy `replicas:sync` does this) |
| Missed history (this tenant’s empty settings) | Watch compares counts and recopies from publisher, then live WAL resumes |
`npm run replicas:check` / `replicas:repair-data` remain for operators; they are
not required for day-to-day imports after the watch is up.
## Local development note