feat(replicas): auto-heal slave data drift and no-store public settings
Add sync-replica-data repair on deploy, a publisher-side replica watch (60s + after tenant/settings writes), REFRESH PUBLICATION after migrate, and Cache-Control no-store on program-settings and get_version. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+59
-5
@@ -122,15 +122,69 @@ cp deploy/replicas.env.example deploy/replicas.env # once; add real DSNs
|
||||
bash deploy/sync-replica-schema.sh --init # once, when schemas already match
|
||||
```
|
||||
|
||||
For every release:
|
||||
For every release on **publisher** (`/var/www/superadmin/api`):
|
||||
|
||||
```bash
|
||||
npm run replicas:sync # DDL to each replica first, then migrate deploy here
|
||||
npm run replicas:check # subscription state + columns missing on any replica
|
||||
git pull
|
||||
npm run replicas:sync # 1) DDL on replicas 2) migrate deploy 3) repair data drift if any
|
||||
npx prisma generate
|
||||
npm run build
|
||||
pm2 restart superadmin-api
|
||||
```
|
||||
|
||||
Keep migrations schema-only. Data backfills would run once per replica and then
|
||||
again through replication, so apply those on the publisher by hand.
|
||||
Check only:
|
||||
|
||||
```bash
|
||||
npm run replicas:check # schema + subscription + row counts (settings, tenants, …)
|
||||
```
|
||||
|
||||
Force data realign from publisher (rare — e.g. after fixing subscriber permissions):
|
||||
|
||||
```bash
|
||||
npm run replicas:repair-data
|
||||
```
|
||||
|
||||
`replicas:sync` runs `sync-replica-data.sh --repair-if-needed` automatically when
|
||||
`settings` / `setting_templates` / `tenants` counts differ from publisher.
|
||||
|
||||
On the **publisher API** (`superadmin-api`), a replica watch also runs: ~8s after
|
||||
boot, then every 60s, and immediately after tenant/settings writes. Drift is
|
||||
healed without an operator command. Logical replication still streams live
|
||||
INSERT/UPDATE/DELETE; the watch only fills **missed history** (WAL is not
|
||||
rewound).
|
||||
|
||||
Each replica API user (`montime` by default) needs `SELECT` on all tables — the
|
||||
repair script re-applies grants. `REPLICA_DSNS` must connect as a role that can
|
||||
`TRUNCATE` + `ALTER SUBSCRIPTION` on the subscriber. The watch is idle unless
|
||||
`deploy/replicas.env` exists on that host.
|
||||
|
||||
Keep migrations schema-only. After each migrate, subscribers get
|
||||
`ALTER SUBSCRIPTION … REFRESH PUBLICATION WITH (copy_data = true)` so **new
|
||||
tables** join replication (Postgres does not add them by itself).
|
||||
|
||||
## 9) Data changes **without** deploy (import settings, edit tenant, …)
|
||||
|
||||
**Normal path:** publisher writes → `superadmin_pub` WAL → slaves apply in
|
||||
**seconds**. No `replicas:sync` for that.
|
||||
|
||||
**Why LSN-healthy slaves can still have empty `settings`:** logical replication
|
||||
is a **stream**, not a mirror. `received_lsn = latest_end_lsn` means “caught up
|
||||
with current WAL”, not “every historical row exists”. Rows inserted while apply
|
||||
was broken, or while the table was **not yet in the subscription**, are never
|
||||
replayed. Postgres will not go back.
|
||||
|
||||
That gap is healed **automatically on the publisher API** (watch every 60s +
|
||||
right after import/edit). You do **not** run `replicas:repair-data` by hand
|
||||
once this build is running on `/var/www/superadmin/api`.
|
||||
|
||||
| Action | What actually happens |
|
||||
|--------|------------------------|
|
||||
| INSERT/UPDATE/DELETE while subscription is ready | Live WAL — slaves update immediately |
|
||||
| New table after `FOR ALL TABLES` publication | Not replicated until `REFRESH PUBLICATION` (deploy `replicas:sync` does this) |
|
||||
| Missed history (this tenant’s empty settings) | Watch compares counts and recopies from publisher, then live WAL resumes |
|
||||
|
||||
`npm run replicas:check` / `replicas:repair-data` remain for operators; they are
|
||||
not required for day-to-day imports after the watch is up.
|
||||
|
||||
## Local development note
|
||||
|
||||
|
||||
Reference in New Issue
Block a user