feat(replicas): auto-heal slave data drift and no-store public settings
Add sync-replica-data repair on deploy, a publisher-side replica watch (60s + after tenant/settings writes), REFRESH PUBLICATION after migrate, and Cache-Control no-store on program-settings and get_version. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+59
-5
@@ -122,15 +122,69 @@ cp deploy/replicas.env.example deploy/replicas.env # once; add real DSNs
|
|||||||
bash deploy/sync-replica-schema.sh --init # once, when schemas already match
|
bash deploy/sync-replica-schema.sh --init # once, when schemas already match
|
||||||
```
|
```
|
||||||
|
|
||||||
For every release:
|
For every release on **publisher** (`/var/www/superadmin/api`):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm run replicas:sync # DDL to each replica first, then migrate deploy here
|
git pull
|
||||||
npm run replicas:check # subscription state + columns missing on any replica
|
npm run replicas:sync # 1) DDL on replicas 2) migrate deploy 3) repair data drift if any
|
||||||
|
npx prisma generate
|
||||||
|
npm run build
|
||||||
|
pm2 restart superadmin-api
|
||||||
```
|
```
|
||||||
|
|
||||||
Keep migrations schema-only. Data backfills would run once per replica and then
|
Check only:
|
||||||
again through replication, so apply those on the publisher by hand.
|
|
||||||
|
```bash
|
||||||
|
npm run replicas:check # schema + subscription + row counts (settings, tenants, …)
|
||||||
|
```
|
||||||
|
|
||||||
|
Force data realign from publisher (rare — e.g. after fixing subscriber permissions):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run replicas:repair-data
|
||||||
|
```
|
||||||
|
|
||||||
|
`replicas:sync` runs `sync-replica-data.sh --repair-if-needed` automatically when
|
||||||
|
`settings` / `setting_templates` / `tenants` counts differ from publisher.
|
||||||
|
|
||||||
|
On the **publisher API** (`superadmin-api`), a replica watch also runs: ~8s after
|
||||||
|
boot, then every 60s, and immediately after tenant/settings writes. Drift is
|
||||||
|
healed without an operator command. Logical replication still streams live
|
||||||
|
INSERT/UPDATE/DELETE; the watch only fills **missed history** (WAL is not
|
||||||
|
rewound).
|
||||||
|
|
||||||
|
Each replica API user (`montime` by default) needs `SELECT` on all tables — the
|
||||||
|
repair script re-applies grants. `REPLICA_DSNS` must connect as a role that can
|
||||||
|
`TRUNCATE` + `ALTER SUBSCRIPTION` on the subscriber. The watch is idle unless
|
||||||
|
`deploy/replicas.env` exists on that host.
|
||||||
|
|
||||||
|
Keep migrations schema-only. After each migrate, subscribers get
|
||||||
|
`ALTER SUBSCRIPTION … REFRESH PUBLICATION WITH (copy_data = true)` so **new
|
||||||
|
tables** join replication (Postgres does not add them by itself).
|
||||||
|
|
||||||
|
## 9) Data changes **without** deploy (import settings, edit tenant, …)
|
||||||
|
|
||||||
|
**Normal path:** publisher writes → `superadmin_pub` WAL → slaves apply in
|
||||||
|
**seconds**. No `replicas:sync` for that.
|
||||||
|
|
||||||
|
**Why LSN-healthy slaves can still have empty `settings`:** logical replication
|
||||||
|
is a **stream**, not a mirror. `received_lsn = latest_end_lsn` means “caught up
|
||||||
|
with current WAL”, not “every historical row exists”. Rows inserted while apply
|
||||||
|
was broken, or while the table was **not yet in the subscription**, are never
|
||||||
|
replayed. Postgres will not go back.
|
||||||
|
|
||||||
|
That gap is healed **automatically on the publisher API** (watch every 60s +
|
||||||
|
right after import/edit). You do **not** run `replicas:repair-data` by hand
|
||||||
|
once this build is running on `/var/www/superadmin/api`.
|
||||||
|
|
||||||
|
| Action | What actually happens |
|
||||||
|
|--------|------------------------|
|
||||||
|
| INSERT/UPDATE/DELETE while subscription is ready | Live WAL — slaves update immediately |
|
||||||
|
| New table after `FOR ALL TABLES` publication | Not replicated until `REFRESH PUBLICATION` (deploy `replicas:sync` does this) |
|
||||||
|
| Missed history (this tenant’s empty settings) | Watch compares counts and recopies from publisher, then live WAL resumes |
|
||||||
|
|
||||||
|
`npm run replicas:check` / `replicas:repair-data` remain for operators; they are
|
||||||
|
not required for day-to-day imports after the watch is up.
|
||||||
|
|
||||||
## Local development note
|
## Local development note
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,9 @@
|
|||||||
# One DSN per line (or comma). Do not put spaces between URLs.
|
# One DSN per line (or comma). Do not put spaces between URLs.
|
||||||
# Prefer URI form. Password must not contain @ : / # ? — or URL-encode it.
|
# Prefer URI form. Password must not contain @ : / # ? — or URL-encode it.
|
||||||
# Keyword form (host=...) is OK only because the script splits on newlines, not spaces.
|
# Keyword form (host=...) is OK only because the script splits on newlines, not spaces.
|
||||||
# The user must be able to run ALTER TABLE (table owner or superuser).
|
# The user must be able to run ALTER TABLE + ALTER SUBSCRIPTION + TRUNCATE on each replica.
|
||||||
REPLICA_DSNS="postgresql://USER:PASSWORD@SLAVE_1_IP:5433/superadmin
|
REPLICA_DSNS="postgresql://USER:PASSWORD@SLAVE_1_IP:5433/superadmin
|
||||||
postgresql://USER:PASSWORD@SLAVE_2_IP:5432/superadmin"
|
postgresql://USER:PASSWORD@SLAVE_2_IP:5432/superadmin"
|
||||||
|
|
||||||
|
# Read-only app user on replicas (program-settings API). Grants re-applied after data repair.
|
||||||
|
# REPLICA_APP_DB_USER=montime
|
||||||
|
|||||||
@@ -0,0 +1,247 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Compare + repair row data on logical-replication subscribers vs publisher.
|
||||||
|
#
|
||||||
|
# replicas:sync only ships DDL. If a subscriber missed initial copy (permissions,
|
||||||
|
# stalled subscription, import before replication caught up), rows stay missing
|
||||||
|
# until repaired. This script makes publisher the source of truth for core tables.
|
||||||
|
#
|
||||||
|
# Usage (on publisher, same deploy/replicas.env as sync-replica-schema.sh):
|
||||||
|
# bash deploy/sync-replica-data.sh --check
|
||||||
|
# bash deploy/sync-replica-data.sh --repair
|
||||||
|
# bash deploy/sync-replica-data.sh --repair-if-needed
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
CONFIG_FILE="${REPLICAS_ENV_FILE:-$API_DIR/deploy/replicas.env}"
|
||||||
|
MODE="${1:---repair-if-needed}"
|
||||||
|
|
||||||
|
# Tables that must match publisher for public API (order: parents before children).
|
||||||
|
SYNC_TABLES=(setting_templates tenants settings)
|
||||||
|
APP_DB_USER="${REPLICA_APP_DB_USER:-montime}"
|
||||||
|
|
||||||
|
case "$MODE" in
|
||||||
|
--check|--repair|--repair-if-needed) ;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 [--check|--repair|--repair-if-needed]" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
command -v psql >/dev/null || { echo "psql not found" >&2; exit 1; }
|
||||||
|
command -v pg_dump >/dev/null || { echo "pg_dump not found" >&2; exit 1; }
|
||||||
|
|
||||||
|
if [[ -f "$CONFIG_FILE" ]]; then
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
set -a && source "$CONFIG_FILE" && set +a
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "${REPLICA_DSNS:-}" ]]; then
|
||||||
|
echo "REPLICA_DSNS is empty. Create $CONFIG_FILE from deploy/replicas.env.example" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "${DATABASE_URL:-}" && -f "$API_DIR/.env" ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
set -a && source "$API_DIR/.env" && set +a
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "${DATABASE_URL:-}" ]]; then
|
||||||
|
echo "DATABASE_URL is empty (publisher)." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mapfile -t REPLICAS < <(
|
||||||
|
printf '%s\n' "$REPLICA_DSNS" | tr ',' '\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | grep -v '^$'
|
||||||
|
)
|
||||||
|
|
||||||
|
run_psql() {
|
||||||
|
psql "$@" </dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
libpq_dsn() {
|
||||||
|
local u="$1"
|
||||||
|
echo "${u%%\?*}"
|
||||||
|
}
|
||||||
|
|
||||||
|
label() {
|
||||||
|
local dsn="$1"
|
||||||
|
if [[ "$dsn" == *"://"* ]]; then
|
||||||
|
sed -E 's#^[a-z]+://([^@]*@)?##; s#\?.*$##' <<< "$dsn"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
local host port db
|
||||||
|
host="$(sed -nE 's/.*host=([^ ]+).*/\1/p' <<< "$dsn")"
|
||||||
|
port="$(sed -nE 's/.*port=([^ ]+).*/\1/p' <<< "$dsn")"
|
||||||
|
db="$(sed -nE 's/.*dbname=([^ ]+).*/\1/p' <<< "$dsn")"
|
||||||
|
echo "${host:-?}:${port:-5432}/${db:-?}"
|
||||||
|
}
|
||||||
|
|
||||||
|
table_count() {
|
||||||
|
local dsn="$1" table="$2"
|
||||||
|
run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
|
||||||
|
"SELECT count(*)::bigint FROM public.\"$table\""
|
||||||
|
}
|
||||||
|
|
||||||
|
subscription_names() {
|
||||||
|
local dsn="$1"
|
||||||
|
run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
|
||||||
|
"SELECT subname FROM pg_subscription ORDER BY subname"
|
||||||
|
}
|
||||||
|
|
||||||
|
grant_app_user() {
|
||||||
|
local dsn="$1"
|
||||||
|
run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -q -c "
|
||||||
|
GRANT USAGE ON SCHEMA public TO ${APP_DB_USER};
|
||||||
|
GRANT SELECT ON ALL TABLES IN SCHEMA public TO ${APP_DB_USER};
|
||||||
|
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO ${APP_DB_USER};
|
||||||
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||||
|
GRANT SELECT ON TABLES TO ${APP_DB_USER};
|
||||||
|
" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
copy_table_from_publisher() {
|
||||||
|
local replica_dsn="$1" table="$2"
|
||||||
|
local pub_dsn
|
||||||
|
pub_dsn="$(libpq_dsn "$DATABASE_URL")"
|
||||||
|
pg_dump "$pub_dsn" \
|
||||||
|
-t "public.\"$table\"" \
|
||||||
|
--data-only \
|
||||||
|
--no-owner \
|
||||||
|
--no-privileges \
|
||||||
|
--disable-triggers \
|
||||||
|
| run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q
|
||||||
|
}
|
||||||
|
|
||||||
|
repair_replica() {
|
||||||
|
local replica_dsn="$1"
|
||||||
|
local lbl drifted=()
|
||||||
|
local table pub replica
|
||||||
|
|
||||||
|
lbl="$(label "$replica_dsn")"
|
||||||
|
echo "== repair $lbl"
|
||||||
|
|
||||||
|
mapfile -t subs < <(subscription_names "$replica_dsn")
|
||||||
|
if ((${#subs[@]} == 0)); then
|
||||||
|
echo " ! no subscription on replica — skip data repair (create subscription first)" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
for table in "${SYNC_TABLES[@]}"; do
|
||||||
|
pub="$(table_count "$DATABASE_URL" "$table")"
|
||||||
|
replica="$(table_count "$replica_dsn" "$table")"
|
||||||
|
if [[ "$pub" != "$replica" ]]; then
|
||||||
|
drifted+=("$table")
|
||||||
|
echo " drift $table publisher=$pub replica=$replica"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ((${#drifted[@]} == 0)); then
|
||||||
|
echo " data already in sync"
|
||||||
|
grant_app_user "$replica_dsn"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# TRUNCATE tenants CASCADE wipes settings — recopy the whole core set.
|
||||||
|
drifted=("${SYNC_TABLES[@]}")
|
||||||
|
|
||||||
|
for sub in "${subs[@]}"; do
|
||||||
|
echo " disabling subscription $sub"
|
||||||
|
run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
|
||||||
|
-c "ALTER SUBSCRIPTION \"${sub}\" DISABLE"
|
||||||
|
done
|
||||||
|
|
||||||
|
for table in "${drifted[@]}"; do
|
||||||
|
echo " truncating + copying $table"
|
||||||
|
run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
|
||||||
|
-c "TRUNCATE TABLE public.\"$table\" CASCADE"
|
||||||
|
copy_table_from_publisher "$replica_dsn" "$table"
|
||||||
|
pub="$(table_count "$DATABASE_URL" "$table")"
|
||||||
|
replica="$(table_count "$replica_dsn" "$table")"
|
||||||
|
if [[ "$pub" != "$replica" ]]; then
|
||||||
|
echo " ! repair failed for $table (publisher=$pub replica=$replica)" >&2
|
||||||
|
for sub in "${subs[@]}"; do
|
||||||
|
run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
|
||||||
|
-c "ALTER SUBSCRIPTION \"${sub}\" ENABLE" || true
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
echo " ok $table=$replica"
|
||||||
|
done
|
||||||
|
|
||||||
|
grant_app_user "$replica_dsn"
|
||||||
|
|
||||||
|
for sub in "${subs[@]}"; do
|
||||||
|
echo " enabling subscription $sub"
|
||||||
|
run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
|
||||||
|
-c "ALTER SUBSCRIPTION \"${sub}\" ENABLE"
|
||||||
|
run_psql "$(libpq_dsn "$replica_dsn")" -v ON_ERROR_STOP=1 -q \
|
||||||
|
-c "ALTER SUBSCRIPTION \"${sub}\" REFRESH PUBLICATION" || true
|
||||||
|
done
|
||||||
|
|
||||||
|
echo " repair done"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_replicas() {
|
||||||
|
local needs_repair=0
|
||||||
|
local dsn lbl table pub replica
|
||||||
|
|
||||||
|
for dsn in "${REPLICAS[@]}"; do
|
||||||
|
lbl="$(label "$dsn")"
|
||||||
|
echo "== $lbl"
|
||||||
|
run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
|
||||||
|
"SELECT subname || ' received=' || COALESCE(received_lsn::text, '-')
|
||||||
|
|| ' latest=' || COALESCE(latest_end_lsn::text, '-')
|
||||||
|
FROM pg_stat_subscription" 2>/dev/null || echo " (no pg_stat_subscription)"
|
||||||
|
|
||||||
|
for table in "${SYNC_TABLES[@]}"; do
|
||||||
|
pub="$(table_count "$DATABASE_URL" "$table")"
|
||||||
|
replica="$(table_count "$dsn" "$table")"
|
||||||
|
if [[ "$pub" == "$replica" ]]; then
|
||||||
|
echo " $table OK ($pub)"
|
||||||
|
else
|
||||||
|
echo " $table DRIFT publisher=$pub replica=$replica"
|
||||||
|
needs_repair=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
done
|
||||||
|
|
||||||
|
return "$needs_repair"
|
||||||
|
}
|
||||||
|
|
||||||
|
LOCK_FILE="${REPLICA_DATA_LOCK:-/tmp/superadmin-replica-data.lock}"
|
||||||
|
if [[ "$MODE" != "--check" ]] && command -v flock >/dev/null; then
|
||||||
|
exec 9>"$LOCK_FILE"
|
||||||
|
if ! flock -n 9; then
|
||||||
|
echo "replica data sync already running — skip"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$MODE" == "--check" ]]; then
|
||||||
|
check_replicas
|
||||||
|
exit $?
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$MODE" == "--repair-if-needed" ]]; then
|
||||||
|
if check_replicas; then
|
||||||
|
echo
|
||||||
|
echo "All replicas in sync."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
echo "Drift detected — repairing..."
|
||||||
|
fi
|
||||||
|
|
||||||
|
failed=0
|
||||||
|
for dsn in "${REPLICAS[@]}"; do
|
||||||
|
repair_replica "$dsn" || failed=1
|
||||||
|
done
|
||||||
|
|
||||||
|
echo
|
||||||
|
if ((failed == 0)); then
|
||||||
|
echo "Replica data sync complete."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Some replicas failed data repair." >&2
|
||||||
|
exit 1
|
||||||
@@ -181,6 +181,30 @@ echo "== publisher"
|
|||||||
cd "$API_DIR"
|
cd "$API_DIR"
|
||||||
npx prisma migrate deploy
|
npx prisma migrate deploy
|
||||||
|
|
||||||
|
refresh_subscriptions() {
|
||||||
|
local dsn="$1"
|
||||||
|
mapfile -t subs < <(
|
||||||
|
run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -At -c \
|
||||||
|
"SELECT subname FROM pg_subscription ORDER BY subname"
|
||||||
|
)
|
||||||
|
if ((${#subs[@]} == 0)); then
|
||||||
|
echo " ! no subscription — skip REFRESH PUBLICATION"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
for sub in "${subs[@]}"; do
|
||||||
|
echo " ALTER SUBSCRIPTION $sub REFRESH PUBLICATION (copy_data)"
|
||||||
|
# New tables added to FOR ALL TABLES are NOT replicated until this runs.
|
||||||
|
run_psql "$(libpq_dsn "$dsn")" -v ON_ERROR_STOP=1 -q \
|
||||||
|
-c "ALTER SUBSCRIPTION \"${sub}\" REFRESH PUBLICATION WITH (copy_data = true)"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "== refresh subscriptions (pick up new tables)"
|
||||||
|
for dsn in "${REPLICAS[@]}"; do
|
||||||
|
echo "== $(label "$dsn")"
|
||||||
|
refresh_subscriptions "$dsn"
|
||||||
|
done
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "Replicas apply DDL first, so the publisher can stream the new columns right away."
|
echo "Replicas apply DDL first, so the publisher can stream the new columns right away."
|
||||||
echo "Verify with: bash deploy/sync-replica-schema.sh --check"
|
echo "Verify with: bash deploy/sync-replica-schema.sh --check"
|
||||||
|
|||||||
+3
-2
@@ -35,8 +35,9 @@
|
|||||||
"prisma:repair-khanegi-ui-config": "ts-node --transpile-only prisma/repair-khanegi-ui-config.ts",
|
"prisma:repair-khanegi-ui-config": "ts-node --transpile-only prisma/repair-khanegi-ui-config.ts",
|
||||||
"prisma:copy-sms-between-tenants": "ts-node --transpile-only prisma/copy-sms-between-tenants.ts",
|
"prisma:copy-sms-between-tenants": "ts-node --transpile-only prisma/copy-sms-between-tenants.ts",
|
||||||
"prisma:list-tenants": "ts-node --transpile-only prisma/list-tenants.ts",
|
"prisma:list-tenants": "ts-node --transpile-only prisma/list-tenants.ts",
|
||||||
"replicas:sync": "bash deploy/sync-replica-schema.sh",
|
"replicas:sync": "bash deploy/sync-replica-schema.sh && bash deploy/sync-replica-data.sh --repair-if-needed",
|
||||||
"replicas:check": "bash deploy/sync-replica-schema.sh --check",
|
"replicas:check": "bash deploy/sync-replica-schema.sh --check && bash deploy/sync-replica-data.sh --check",
|
||||||
|
"replicas:repair-data": "bash deploy/sync-replica-data.sh --repair",
|
||||||
"settings:seed-templates": "echo 'Templates seed on first GET /api/setting-templates or POST /api/setting-templates/seed'"
|
"settings:seed-templates": "echo 'Templates seed on first GET /api/setting-templates or POST /api/setting-templates/seed'"
|
||||||
},
|
},
|
||||||
"prisma": {
|
"prisma": {
|
||||||
|
|||||||
@@ -16,10 +16,12 @@ import { SupportModule } from './support/support.module';
|
|||||||
import { NotificationsModule } from './notifications/notifications.module';
|
import { NotificationsModule } from './notifications/notifications.module';
|
||||||
import { AdminsModule } from './admins/admins.module';
|
import { AdminsModule } from './admins/admins.module';
|
||||||
import { HealthController } from './health.controller';
|
import { HealthController } from './health.controller';
|
||||||
|
import { ReplicaWatchModule } from './replica-watch/replica-watch.module';
|
||||||
|
|
||||||
@Module({
|
@Module({
|
||||||
imports: [
|
imports: [
|
||||||
ConfigModule.forRoot({ isGlobal: true }),
|
ConfigModule.forRoot({ isGlobal: true }),
|
||||||
|
ReplicaWatchModule,
|
||||||
PrismaModule,
|
PrismaModule,
|
||||||
NotificationsModule,
|
NotificationsModule,
|
||||||
AuthModule,
|
AuthModule,
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { Controller, Get, Headers, Query } from '@nestjs/common';
|
import { Controller, Get, Header, Headers, Query } from '@nestjs/common';
|
||||||
import { GetVersionService } from './get-version.service';
|
import { GetVersionService } from './get-version.service';
|
||||||
|
|
||||||
@Controller()
|
@Controller()
|
||||||
@@ -6,6 +6,7 @@ export class GetVersionController {
|
|||||||
constructor(private readonly getVersionService: GetVersionService) {}
|
constructor(private readonly getVersionService: GetVersionService) {}
|
||||||
|
|
||||||
@Get('get_version')
|
@Get('get_version')
|
||||||
|
@Header('Cache-Control', 'no-store')
|
||||||
getVersion(
|
getVersion(
|
||||||
@Headers('host') host?: string,
|
@Headers('host') host?: string,
|
||||||
@Headers('x-forwarded-host') forwardedHost?: string,
|
@Headers('x-forwarded-host') forwardedHost?: string,
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ export class ApiPublicProgramSettingsController {
|
|||||||
constructor(private readonly programSettings: ProgramSettingsService) {}
|
constructor(private readonly programSettings: ProgramSettingsService) {}
|
||||||
|
|
||||||
@Get()
|
@Get()
|
||||||
@Header('Cache-Control', 'public, max-age=30')
|
@Header('Cache-Control', 'no-store')
|
||||||
getSettings(
|
getSettings(
|
||||||
@Headers('host') host?: string,
|
@Headers('host') host?: string,
|
||||||
@Headers('x-forwarded-host') forwardedHost?: string,
|
@Headers('x-forwarded-host') forwardedHost?: string,
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import { join } from 'path';
|
|||||||
import { Prisma, ProductType, Setting, SettingTemplate } from '@prisma/client';
|
import { Prisma, ProductType, Setting, SettingTemplate } from '@prisma/client';
|
||||||
import { PrismaService } from '../prisma/prisma.service';
|
import { PrismaService } from '../prisma/prisma.service';
|
||||||
import { AuditService } from '../audit/audit.service';
|
import { AuditService } from '../audit/audit.service';
|
||||||
|
import { ReplicaWatchService } from '../replica-watch/replica-watch.service';
|
||||||
import { isSecretOrServerOnlySettingKey } from './secret-setting-keys';
|
import { isSecretOrServerOnlySettingKey } from './secret-setting-keys';
|
||||||
import { isRuntimeProgramSettingKey } from './runtime-setting-keys';
|
import { isRuntimeProgramSettingKey } from './runtime-setting-keys';
|
||||||
import { DEPLOY_SETTING_CATALOG_KEYS } from './deploy-setting-catalog-keys';
|
import { DEPLOY_SETTING_CATALOG_KEYS } from './deploy-setting-catalog-keys';
|
||||||
@@ -57,6 +58,7 @@ export class ProgramSettingsService implements OnModuleInit {
|
|||||||
constructor(
|
constructor(
|
||||||
private readonly prisma: PrismaService,
|
private readonly prisma: PrismaService,
|
||||||
private readonly audit: AuditService,
|
private readonly audit: AuditService,
|
||||||
|
private readonly replicaWatch: ReplicaWatchService,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
async onModuleInit() {
|
async onModuleInit() {
|
||||||
@@ -173,6 +175,7 @@ export class ProgramSettingsService implements OnModuleInit {
|
|||||||
});
|
});
|
||||||
|
|
||||||
const items = await this.listTenantRows(tenantId);
|
const items = await this.listTenantRows(tenantId);
|
||||||
|
this.replicaWatch.kick();
|
||||||
return {
|
return {
|
||||||
tenant: this.tenantMeta(tenant),
|
tenant: this.tenantMeta(tenant),
|
||||||
items,
|
items,
|
||||||
@@ -320,6 +323,7 @@ export class ProgramSettingsService implements OnModuleInit {
|
|||||||
});
|
});
|
||||||
|
|
||||||
const items = await this.listTenantRows(tenantId);
|
const items = await this.listTenantRows(tenantId);
|
||||||
|
this.replicaWatch.kick();
|
||||||
return {
|
return {
|
||||||
tenant: this.tenantMeta(tenant),
|
tenant: this.tenantMeta(tenant),
|
||||||
apiBase,
|
apiBase,
|
||||||
@@ -390,6 +394,7 @@ export class ProgramSettingsService implements OnModuleInit {
|
|||||||
after: { keys: incoming.map((r) => r.key), tenants: tenants.length } as Prisma.InputJsonValue,
|
after: { keys: incoming.map((r) => r.key), tenants: tenants.length } as Prisma.InputJsonValue,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
this.replicaWatch.kick();
|
||||||
return {
|
return {
|
||||||
keys: incoming.map((r) => r.key),
|
keys: incoming.map((r) => r.key),
|
||||||
tenantsUpdated: tenants.length,
|
tenantsUpdated: tenants.length,
|
||||||
@@ -416,6 +421,7 @@ export class ProgramSettingsService implements OnModuleInit {
|
|||||||
after: result as Prisma.InputJsonValue,
|
after: result as Prisma.InputJsonValue,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
this.replicaWatch.kick();
|
||||||
return { key, ...result };
|
return { key, ...result };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ export class PublicProgramSettingsController {
|
|||||||
constructor(private readonly programSettings: ProgramSettingsService) {}
|
constructor(private readonly programSettings: ProgramSettingsService) {}
|
||||||
|
|
||||||
@Get('get_settings')
|
@Get('get_settings')
|
||||||
@Header('Cache-Control', 'public, max-age=30')
|
@Header('Cache-Control', 'no-store')
|
||||||
getSettings(
|
getSettings(
|
||||||
@Headers('host') host?: string,
|
@Headers('host') host?: string,
|
||||||
@Headers('x-forwarded-host') forwardedHost?: string,
|
@Headers('x-forwarded-host') forwardedHost?: string,
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
import { Global, Module } from '@nestjs/common';
|
||||||
|
import { ReplicaWatchService } from './replica-watch.service';
|
||||||
|
|
||||||
|
@Global()
|
||||||
|
@Module({
|
||||||
|
providers: [ReplicaWatchService],
|
||||||
|
exports: [ReplicaWatchService],
|
||||||
|
})
|
||||||
|
export class ReplicaWatchModule {}
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
import { Injectable, Logger, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
|
||||||
|
import { spawn } from 'child_process';
|
||||||
|
import { existsSync } from 'fs';
|
||||||
|
import { join } from 'path';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Logical replication streams WAL, it does not re-copy missed history.
|
||||||
|
* If a replica drifted (new table never REFRESH'd, apply error, …) this
|
||||||
|
* process heals it on the publisher — no operator command required.
|
||||||
|
*/
|
||||||
|
@Injectable()
|
||||||
|
export class ReplicaWatchService implements OnModuleInit, OnModuleDestroy {
|
||||||
|
private readonly logger = new Logger(ReplicaWatchService.name);
|
||||||
|
private timer?: ReturnType<typeof setInterval>;
|
||||||
|
private running = false;
|
||||||
|
private pending = false;
|
||||||
|
|
||||||
|
onModuleInit() {
|
||||||
|
if (!this.enabled()) {
|
||||||
|
this.logger.log('replica watch idle (no deploy/replicas.env on this host)');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.logger.log('replica watch on — drift auto-repairs every 60s');
|
||||||
|
this.timer = setInterval(() => this.kick(), 60_000);
|
||||||
|
setTimeout(() => this.kick(), 8_000);
|
||||||
|
}
|
||||||
|
|
||||||
|
onModuleDestroy() {
|
||||||
|
if (this.timer) clearInterval(this.timer);
|
||||||
|
}
|
||||||
|
|
||||||
|
kick() {
|
||||||
|
if (!this.enabled()) return;
|
||||||
|
if (this.running) {
|
||||||
|
this.pending = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
void this.run();
|
||||||
|
}
|
||||||
|
|
||||||
|
private enabled() {
|
||||||
|
return existsSync(this.envFile()) && existsSync(this.scriptPath());
|
||||||
|
}
|
||||||
|
|
||||||
|
private envFile() {
|
||||||
|
return join(process.cwd(), 'deploy/replicas.env');
|
||||||
|
}
|
||||||
|
|
||||||
|
private scriptPath() {
|
||||||
|
return join(process.cwd(), 'deploy/sync-replica-data.sh');
|
||||||
|
}
|
||||||
|
|
||||||
|
private run(): Promise<void> {
|
||||||
|
this.running = true;
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
const child = spawn('bash', [this.scriptPath(), '--repair-if-needed'], {
|
||||||
|
cwd: process.cwd(),
|
||||||
|
env: process.env,
|
||||||
|
});
|
||||||
|
let out = '';
|
||||||
|
child.stdout?.on('data', (buf: Buffer) => {
|
||||||
|
out += buf.toString();
|
||||||
|
});
|
||||||
|
child.stderr?.on('data', (buf: Buffer) => {
|
||||||
|
out += buf.toString();
|
||||||
|
});
|
||||||
|
child.on('error', (err) => {
|
||||||
|
this.logger.warn(`replica watch failed to start: ${err.message}`);
|
||||||
|
this.finish();
|
||||||
|
resolve();
|
||||||
|
});
|
||||||
|
child.on('close', (code) => {
|
||||||
|
const text = out.trim();
|
||||||
|
const healed = /Drift detected|repair done|Replica data sync complete/i.test(text);
|
||||||
|
if (code !== 0) {
|
||||||
|
this.logger.warn(`replica watch exit ${code}\n${text.slice(-2000)}`);
|
||||||
|
} else if (healed) {
|
||||||
|
this.logger.log(text.slice(-2000) || 'replica data repaired');
|
||||||
|
}
|
||||||
|
this.finish();
|
||||||
|
resolve();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private finish() {
|
||||||
|
this.running = false;
|
||||||
|
if (this.pending) {
|
||||||
|
this.pending = false;
|
||||||
|
void this.run();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,6 +8,7 @@ import { PrismaService } from '../prisma/prisma.service';
|
|||||||
import { AuditService } from '../audit/audit.service';
|
import { AuditService } from '../audit/audit.service';
|
||||||
import { AdminNotifyService } from '../notifications/admin-notify.service';
|
import { AdminNotifyService } from '../notifications/admin-notify.service';
|
||||||
import { ProgramSettingsService } from '../program-settings/program-settings.service';
|
import { ProgramSettingsService } from '../program-settings/program-settings.service';
|
||||||
|
import { ReplicaWatchService } from '../replica-watch/replica-watch.service';
|
||||||
import { CreateTenantDto } from './dto/create-tenant.dto';
|
import { CreateTenantDto } from './dto/create-tenant.dto';
|
||||||
import { UpdateTenantDto } from './dto/update-tenant.dto';
|
import { UpdateTenantDto } from './dto/update-tenant.dto';
|
||||||
|
|
||||||
@@ -18,6 +19,7 @@ export class TenantsService {
|
|||||||
private readonly audit: AuditService,
|
private readonly audit: AuditService,
|
||||||
private readonly notify: AdminNotifyService,
|
private readonly notify: AdminNotifyService,
|
||||||
private readonly programSettings: ProgramSettingsService,
|
private readonly programSettings: ProgramSettingsService,
|
||||||
|
private readonly replicaWatch: ReplicaWatchService,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
async findAll(product?: ProductType) {
|
async findAll(product?: ProductType) {
|
||||||
@@ -124,6 +126,7 @@ export class TenantsService {
|
|||||||
`سوپرادمین: نسخه جدید «${tenant.nameFa}» (${tenant.domain}) ایجاد شد.`,
|
`سوپرادمین: نسخه جدید «${tenant.nameFa}» (${tenant.domain}) ایجاد شد.`,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
this.replicaWatch.kick();
|
||||||
return this.sanitize(tenant);
|
return this.sanitize(tenant);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.handleUnique(error);
|
this.handleUnique(error);
|
||||||
@@ -245,6 +248,7 @@ export class TenantsService {
|
|||||||
: `سوپرادمین: نسخه «${tenant.nameFa}» بهروزرسانی شد.`,
|
: `سوپرادمین: نسخه «${tenant.nameFa}» بهروزرسانی شد.`,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
this.replicaWatch.kick();
|
||||||
return this.sanitize(tenant);
|
return this.sanitize(tenant);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.handleUnique(error);
|
this.handleUnique(error);
|
||||||
@@ -273,6 +277,7 @@ export class TenantsService {
|
|||||||
`سوپرادمین: نسخه «${before.nameFa}» (${before.domain}) حذف شد.`,
|
`سوپرادمین: نسخه «${before.nameFa}» (${before.domain}) حذف شد.`,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
this.replicaWatch.kick();
|
||||||
return { ok: true };
|
return { ok: true };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user