feat(replicas): auto-heal slave data drift and no-store public settings

Add sync-replica-data repair on deploy, a publisher-side replica watch (60s + after tenant/settings writes), REFRESH PUBLICATION after migrate, and Cache-Control no-store on program-settings and get_version. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-18 12:32:30 +03:30
parent 668bbcc1db
commit 16d562caa5
13 changed files with 456 additions and 11 deletions
@@ -122,15 +122,69 @@ cp deploy/replicas.env.example deploy/replicas.env   # once; add real DSNs
 bash deploy/sync-replica-schema.sh --init            # once, when schemas already match
 ```

-For every release:
+For every release on **publisher** (`/var/www/superadmin/api`):

 ```bash
-npm run replicas:sync    # DDL to each replica first, then migrate deploy here
-npm run replicas:check   # subscription state + columns missing on any replica
+git pull
+npm run replicas:sync      # 1) DDL on replicas  2) migrate deploy  3) repair data drift if any
+npx prisma generate
+npm run build
+pm2 restart superadmin-api
 ```

-Keep migrations schema-only. Data backfills would run once per replica and then
-again through replication, so apply those on the publisher by hand.
+Check only:
+
+```bash
+npm run replicas:check     # schema + subscription + row counts (settings, tenants, …)
+```
+
+Force data realign from publisher (rare — e.g. after fixing subscriber permissions):
+
+```bash
+npm run replicas:repair-data
+```
+
+`replicas:sync` runs `sync-replica-data.sh --repair-if-needed` automatically when
+`settings` / `setting_templates` / `tenants` counts differ from publisher.
+
+On the **publisher API** (`superadmin-api`), a replica watch also runs: ~8s after
+boot, then every 60s, and immediately after tenant/settings writes. Drift is
+healed without an operator command. Logical replication still streams live
+INSERT/UPDATE/DELETE; the watch only fills **missed history** (WAL is not
+rewound).
+
+Each replica API user (`montime` by default) needs `SELECT` on all tables — the
+repair script re-applies grants. `REPLICA_DSNS` must connect as a role that can
+`TRUNCATE` + `ALTER SUBSCRIPTION` on the subscriber. The watch is idle unless
+`deploy/replicas.env` exists on that host.
+
+Keep migrations schema-only. After each migrate, subscribers get
+`ALTER SUBSCRIPTION … REFRESH PUBLICATION WITH (copy_data = true)` so **new
+tables** join replication (Postgres does not add them by itself).
+
+## 9) Data changes **without** deploy (import settings, edit tenant, …)
+
+**Normal path:** publisher writes → `superadmin_pub` WAL → slaves apply in
+**seconds**. No `replicas:sync` for that.
+
+**Why LSN-healthy slaves can still have empty `settings`:** logical replication
+is a **stream**, not a mirror. `received_lsn = latest_end_lsn` means “caught up
+with current WAL”, not “every historical row exists”. Rows inserted while apply
+was broken, or while the table was **not yet in the subscription**, are never
+replayed. Postgres will not go back.
+
+That gap is healed **automatically on the publisher API** (watch every 60s +
+right after import/edit). You do **not** run `replicas:repair-data` by hand
+once this build is running on `/var/www/superadmin/api`.
+
+| Action | What actually happens |
+|--------|------------------------|
+| INSERT/UPDATE/DELETE while subscription is ready | Live WAL — slaves update immediately |
+| New table after `FOR ALL TABLES` publication | Not replicated until `REFRESH PUBLICATION` (deploy `replicas:sync` does this) |
+| Missed history (this tenant’s empty settings) | Watch compares counts and recopies from publisher, then live WAL resumes |
+
+`npm run replicas:check` / `replicas:repair-data` remain for operators; they are
+not required for day-to-day imports after the watch is up.

 ## Local development note

@@ -3,6 +3,9 @@
 # One DSN per line (or comma). Do not put spaces between URLs.
 # Prefer URI form. Password must not contain @ : / # ? — or URL-encode it.
 # Keyword form (host=...) is OK only because the script splits on newlines, not spaces.
-# The user must be able to run ALTER TABLE (table owner or superuser).
+# The user must be able to run ALTER TABLE + ALTER SUBSCRIPTION + TRUNCATE on each replica.
 REPLICA_DSNS="postgresql://USER:PASSWORD@SLAVE_1_IP:5433/superadmin
 postgresql://USER:PASSWORD@SLAVE_2_IP:5432/superadmin"
+
+# Read-only app user on replicas (program-settings API). Grants re-applied after data repair.
+# REPLICA_APP_DB_USER=montime
@@ -0,0 +1,247 @@
+#!/usr/bin/env bash
+# Compare + repair row data on logical-replication subscribers vs publisher.
+#
+# replicas:sync only ships DDL. If a subscriber missed initial copy (permissions,
+# stalled subscription, import before replication caught up), rows stay missing
+# until repaired. This script makes publisher the source of truth for core tables.
+#
+# Usage (on publisher, same deploy/replicas.env as sync-replica-schema.sh):
+#   bash deploy/sync-replica-data.sh --check
+#   bash deploy/sync-replica-data.sh --repair
+#   bash deploy/sync-replica-data.sh --repair-if-needed
+set -euo pipefail
+
+API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+CONFIG_FILE="${REPLICAS_ENV_FILE:-$API_DIR/deploy/replicas.env}"
+MODE="${1:---repair-if-needed}"
+
+# Tables that must match publisher for public API (order: parents before children).
+SYNC_TABLES=(setting_templates tenants settings)
+APP_DB_USER="${REPLICA_APP_DB_USER:-montime}"
+
+case "$MODE" in
+  --check|--repair|--repair-if-needed) ;;
+  *)
+    echo "Usage: $0 [--check|--repair|--repair-if-needed]" >&2
+    exit 1
+    ;;
+esac
+
+command -v psql >/dev/null || { echo "psql not found" >&2; exit 1; }
+command -v pg_dump >/dev/null || { echo "pg_dump not found" >&2; exit 1; }
+
+if [[ -f "$CONFIG_FILE" ]]; then
+  # shellcheck disable=SC1090
+  set -a && source "$CONFIG_FILE" && set +a
+fi
+
+if [[ -z "${REPLICA_DSNS:-}" ]]; then
+  echo "REPLICA_DSNS is empty. Create $CONFIG_FILE from deploy/replicas.env.example" >&2
+  exit 1
+fi
+
+if [[ -z "${DATABASE_URL:-}" && -f "$API_DIR/.env" ]]; then
+  # shellcheck disable=SC1091
+  set -a && source "$API_DIR/.env" && set +a
+fi
+
+if [[ -z "${DATABASE_URL:-}" ]]; then
+  echo "DATABASE_URL is empty (publisher)." >&2
+  exit 1
+fi
+
+mapfile -t REPLICAS < <(
+  printf '%s\n' "$REPLICA_DSNS" | tr ',' '\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | grep -v '^$'
+)
+
+run_psql() {
+  psql "$@" </dev/null
+}
+
+libpq_dsn() {
+  local u="$1"
+  echo "${u%%\?*}"
+}
+
+label() {
+  local dsn="$1"
+  if [[ "$dsn" == *"://"* ]]; then
+    sed -E 's#^[a-z]+://([^@]*@)?##; s#\?.*$##' <<< "$dsn"
+    return
+  fi
+  local host port db
+  host="$(sed -nE 's/.*host=([^ ]+).*/\1/p' <<< "$dsn")"
+  port="$(sed -nE 's/.*port=([^ ]+).*/\1/p' <<< "$dsn")"
+  db="$(sed -nE 's/.*dbname=([^ ]+).*/\1/p' <<< "$dsn")"
+  echo "${host:-?}:${port:-5432}/${db:-?}"
+}
+
+table_count() {
+  local dsn="$1" table="$2"
+  run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
+    "SELECT count(*)::bigint FROM public.\"$table\""
+}
+
+subscription_names() {
+  local dsn="$1"
+  run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
+    "SELECT subname FROM pg_subscription ORDER BY subname"
+}
+
+grant_app_user() {
+  local dsn="$1"
+  run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -q -c "
+    GRANT USAGE ON SCHEMA public TO ${APP_DB_USER};
+    GRANT SELECT ON ALL TABLES IN SCHEMA public TO ${APP_DB_USER};
+    GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO ${APP_DB_USER};
+    ALTER DEFAULT PRIVILEGES IN SCHEMA public
+      GRANT SELECT ON TABLES TO ${APP_DB_USER};
+  " 2>/dev/null || true
+}
+
+copy_table_from_publisher() {
+  local replica_dsn="$1" table="$2"
+  local pub_dsn
+  pub_dsn="$(libpq_dsn "$DATABASE_URL")"
+  pg_dump "$pub_dsn" \
+    -t "public.\"$table\"" \
+    --data-only \
+    --no-owner \
+    --no-privileges \
+    --disable-triggers \
+    | run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q
+}
+
+repair_replica() {
+  local replica_dsn="$1"
+  local lbl drifted=()
+  local table pub replica
+
+  lbl="$(label "$replica_dsn")"
+  echo "== repair $lbl"
+
+  mapfile -t subs < <(subscription_names "$replica_dsn")
+  if ((${#subs[@]} == 0)); then
+    echo "  ! no subscription on replica — skip data repair (create subscription first)" >&2
+    return 1
+  fi
+
+  for table in "${SYNC_TABLES[@]}"; do
+    pub="$(table_count "$DATABASE_URL" "$table")"
+    replica="$(table_count "$replica_dsn" "$table")"
+    if [[ "$pub" != "$replica" ]]; then
+      drifted+=("$table")
+      echo "  drift $table publisher=$pub replica=$replica"
+    fi
+  done
+
+  if ((${#drifted[@]} == 0)); then
+    echo "  data already in sync"
+    grant_app_user "$replica_dsn"
+    return 0
+  fi
+
+  # TRUNCATE tenants CASCADE wipes settings — recopy the whole core set.
+  drifted=("${SYNC_TABLES[@]}")
+
+  for sub in "${subs[@]}"; do
+    echo "  disabling subscription $sub"
+    run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
+      -c "ALTER SUBSCRIPTION \"${sub}\" DISABLE"
+  done
+
+  for table in "${drifted[@]}"; do
+    echo "  truncating + copying $table"
+    run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
+      -c "TRUNCATE TABLE public.\"$table\" CASCADE"
+    copy_table_from_publisher "$replica_dsn" "$table"
+    pub="$(table_count "$DATABASE_URL" "$table")"
+    replica="$(table_count "$replica_dsn" "$table")"
+    if [[ "$pub" != "$replica" ]]; then
+      echo "  ! repair failed for $table (publisher=$pub replica=$replica)" >&2
+      for sub in "${subs[@]}"; do
+        run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
+          -c "ALTER SUBSCRIPTION \"${sub}\" ENABLE" || true
+      done
+      return 1
+    fi
+    echo "  ok $table=$replica"
+  done
+
+  grant_app_user "$replica_dsn"
+
+  for sub in "${subs[@]}"; do
+    echo "  enabling subscription $sub"
+    run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
+      -c "ALTER SUBSCRIPTION \"${sub}\" ENABLE"
+    run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
+      -c "ALTER SUBSCRIPTION \"${sub}\" REFRESH PUBLICATION" || true
+  done
+
+  echo "  repair done"
+}
+
+check_replicas() {
+  local needs_repair=0
+  local dsn lbl table pub replica
+
+  for dsn in "${REPLICAS[@]}"; do
+    lbl="$(label "$dsn")"
+    echo "== $lbl"
+    run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
+      "SELECT subname || '  received=' || COALESCE(received_lsn::text, '-')
+              || '  latest=' || COALESCE(latest_end_lsn::text, '-')
+        FROM pg_stat_subscription" 2>/dev/null || echo "  (no pg_stat_subscription)"
+
+    for table in "${SYNC_TABLES[@]}"; do
+      pub="$(table_count "$DATABASE_URL" "$table")"
+      replica="$(table_count "$dsn" "$table")"
+      if [[ "$pub" == "$replica" ]]; then
+        echo "  $table OK ($pub)"
+      else
+        echo "  $table DRIFT publisher=$pub replica=$replica"
+        needs_repair=1
+      fi
+    done
+  done
+
+  return "$needs_repair"
+}
+
+LOCK_FILE="${REPLICA_DATA_LOCK:-/tmp/superadmin-replica-data.lock}"
+if [[ "$MODE" != "--check" ]] && command -v flock >/dev/null; then
+  exec 9>"$LOCK_FILE"
+  if ! flock -n 9; then
+    echo "replica data sync already running — skip"
+    exit 0
+  fi
+fi
+
+if [[ "$MODE" == "--check" ]]; then
+  check_replicas
+  exit $?
+fi
+
+if [[ "$MODE" == "--repair-if-needed" ]]; then
+  if check_replicas; then
+    echo
+    echo "All replicas in sync."
+    exit 0
+  fi
+  echo
+  echo "Drift detected — repairing..."
+fi
+
+failed=0
+for dsn in "${REPLICAS[@]}"; do
+  repair_replica "$dsn" || failed=1
+done
+
+echo
+if ((failed == 0)); then
+  echo "Replica data sync complete."
+  exit 0
+fi
+
+echo "Some replicas failed data repair." >&2
+exit 1
@@ -181,6 +181,30 @@ echo "== publisher"
 cd "$API_DIR"
 npx prisma migrate deploy

+refresh_subscriptions() {
+  local dsn="$1"
+  mapfile -t subs < <(
+    run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
+      "SELECT subname FROM pg_subscription ORDER BY subname"
+  )
+  if ((${#subs[@]} == 0)); then
+    echo "  ! no subscription — skip REFRESH PUBLICATION"
+    return 0
+  fi
+  for sub in "${subs[@]}"; do
+    echo "  ALTER SUBSCRIPTION $sub REFRESH PUBLICATION (copy_data)"
+    # New tables added to FOR ALL TABLES are NOT replicated until this runs.
+    run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -q \
+      -c "ALTER SUBSCRIPTION \"${sub}\" REFRESH PUBLICATION WITH (copy_data = true)"
+  done
+}
+
+echo "== refresh subscriptions (pick up new tables)"
+for dsn in "${REPLICAS[@]}"; do
+  echo "== $(label "$dsn")"
+  refresh_subscriptions "$dsn"
+done
+
 echo
 echo "Replicas apply DDL first, so the publisher can stream the new columns right away."
 echo "Verify with: bash deploy/sync-replica-schema.sh --check"