Status: SCHEDULED, NOT EXECUTED — window Sunday 2026-10-18, 02:00–04:00 Europe/London
(authorised 2026-09-14; date agreed by Matthew on 2026-10-13). Chosen for lowest traffic, with a
rollback still possible before Monday business. The earlier proposed date (2026-09-20) lapsed.
Inventory re-verified 2026-10-13 (read-only; values never printed). Supersedes the raw research
page now archived at /en/archive/rotating_password, whose central premise — that PostgreSQL accepts
two passwords for one role at once — is not true on this platform (see Verified facts).
Three corrections, 2026-10-15 — all about the connection pooler, and two of them correct me. The "no
connection pooler in the estate" premise below is false (one runs on the app engine, outside every app's request
path); its stored server credentials for two databases carry the rotating literal (so they must be
regenerated as part of the window, not only after it); and I was wrong twice over about the pooler's
credential — see the correction below, which supersedes the "are wrong / do not authenticate" line this block
first carried. The window's mechanism is unchanged. See Verified facts and Execution pack.
The central server is PostgreSQL 15.18. A role has exactly one stored verifier
(pg_authid.rolpassword), so there is no dual-password rolling cutover. The only two mechanisms are:
(a) create a parallel login role and cut over to it, or (b) an in-place ALTER ROLE … PASSWORD
inside a maintenance window. This runbook uses (b), with the window announced, because (a) also changes
the role name in every connection string and needs the grants/ownership replicated.
Corrected 2026-10-15 — the line here used to read "and there is no connection pooler in the estate (no
pgBouncer / Pgpool container on any host)", and that is now false. A pgbouncer container
(gp_booking_pgbouncer, image edoburu/pgbouncer) runs on the app engine, listening on 4321, with its
own config directory /etc/veripath/pgbouncer/ (pgbouncer.ini + userlist.txt, both 0600). It is not in
any app's request path (no app container's DB_HOST points at it — it serves the scalability rig), which is
why the two findings below are dormant rather than an outage.
Its stored server credentials are the literal, measured 2026-10-15 (session 82) — and my earlier reading of
this file was wrong in two ways. Corrected here rather than quietly dropped, because this page is evidence:
SELECT 1 succeeds for dsp_clinic andveripath_dental (it makes the server login itself, so the test host is irrelevant). Over TCP from the app+2 was the surrounding quotes. The twouser=postgres entries for dsp_clinic and veripath_dental are hash-identical to the app's own currentscan-literal.sh reports at/etc/.../pgbouncer/pgbouncer.ini:3 and :7. The other three entries hold that tenant's/instance's owntest_client_dental,test_client_pharmacy) sit on paths that accept any password — a deliberately wrong value connects — somd5 entry, auth_type=md5) is the one whose rotation was owed from theThe credential in use is a single 8-character literal shared by four different secrets:
POSTGRES_PASSWORD (the superuser the apps log in as), WIKI_DB_PASSWORD, REDIS_PASSWORD
(GP / dental / pharmacy), and the tenants' db_password / CLIENT_DB_PASSWORD.
Occurrences measured 2026-10-13 by two independent read-only passes. Both metrics are given because
they count different things (a file can hold the value several times, and the second pass covered roots
the first did not):
| Host | Files carrying the value | Credential occurrences (line count) |
|---|---|---|
| Primary | 33 | 45 |
| App engine | 13 | 23 |
| Auxiliary | (no scan path from here — see §Auxiliary) | 8 + 1 live-site file |
Eight characters is below current guidance on its own, independent of the sharing. The primary figure
fell from 36 files to 33 on 2026-10-13: six Hermes profile skill scripts were changed to read the
canonical config (−6), while the app-engine side's findings added two /usr/local/bin scripts and the
rotation pack's own old-value carrier (+3, both intentional until the window closes).
Two role facts that shape the window (verified 2026-10-13): the central fh_app_user owns no
objects and has no consumer, so it is safe to alter and is not a literal consumer; the client-tier
fh_app_user likewise rejects the literal over TCP and is not rotated (its real password replaces the
stale literal in the dev envs). The tenancy rows for gp and dental do hold the literal, as
postgres@localhost:5432, and are updated through Django (Fernet) — never raw SQL.
No git exposure: config/secrets/ is gitignored (.gitignore:6) and
git log --all -S<value> returns 0 hits in the gp / dental / pharmacy repos. No history
rewrite is needed for this credential — unlike the 2026-08-15 secret-scan finding, which recorded
it as committed (the repos appear to have been scrubbed since).
Least privilege, separate item: the applications connect as the postgres superuser. A
dedicated application role is a DSPT 4.1.1 / 8.1.4 improvement in its own right; it is out of
scope here but worth doing while the window is open.
Because the same literal fills four secrets, a rotation that changes only the Postgres superuser
would silently break Wiki.js, Redis and every tenant row. Do one of:
Approach chosen 2026-10-13: option 1 — decouple the shared value first, then rotate the superuser.
(Recommendation adopted as the working decision; reversible if the single-window variant is preferred.)
Execution is the app-engine side's; the window is Sunday 2026-10-18, 02:00–04:00 Europe/London.
| Location | Key(s) |
|---|---|
/opt/infra/.env |
POSTGRES_PASSWORD, WIKI_DB_PASSWORD |
/opt/wagtail/docker-compose.yml |
DB_PASS (hardcoded) |
/opt/cold-standby/docker-compose.yml |
POSTGRES_PASSWORD, DB_PASSWORD ×2 (hardcoded) |
/etc/default/gp-booking-celery |
DB_PASSWORD |
/root/.config/veripath-db.conf |
password, central.password, wikijs.password |
gp_booking_app/config/secrets/env.{dev,production} + .env |
DB_PASSWORD, CLIENT_DB_PASSWORD, REDIS_PASSWORD |
dental_booking_app/config/secrets/env.{dev,production} + .env |
as above |
pharmacy_booking_app/config/secrets/env.production |
DB_PASSWORD |
erpnext-deploy/.env, config/common_site_config.json |
REDIS_PASSWORD, redis URLs |
/opt/security-scanner-scripts/backup_wiki.sh.archived |
PG_PASS |
/usr/local/bin/preflight-backup.sh, /usr/local/bin/post-restart-verify.sh |
PGPASSWORD (hardcoded; found 2026-10-13 by the app-engine side) |
Hermes profile skill scripts (6 files under ~/.hermes/profiles/{docs,demo,marketing}-agent/skills/veripath/veripath-wiki-assets/scripts/) |
Fixed 2026-10-13 — now read the canonical config at run time; no literal remains, so nothing further is needed here during the window |
| Stale copies to purge | /opt/__pycache__/*.pyc, /root/gp_booking_app_dev/config/secrets/*, /root/VPS_SETUP_SUMMARY.md, /root/cleanup-quarantine-20260811/backup-client.py, /root/backups/gp_booking_app_dev-uncommitted-20260911.patch, gp_booking_app_backup_20260707_130039/ (README.md, start_celery_worker.sh, scripts/migrate_env_settings.sh, tenancy/migrations/0002_seed_sectors_and_tenants.py) |
| Documents that quote the literal and must be redacted | /root/forensics/secret-scan-2026-08-15.md, /root/work/hosting-provider-response-2026-08-20.md |
| Agent session transcripts | /root/.hermes/sessions/request_dump_*.json (contain older tool output) |
Central DB tenancy_tenant rows |
db_password (encrypted; GP / dental / pharmacy) |
| Location | Key(s) |
|---|---|
gp_booking_app/dsp_clinic/settings.py (~line 229) |
CHANNEL_LAYER_URL hardcoded default |
gp_booking_app/tenancy/migrations/0002_seed_sectors_and_tenants.py |
plaintext db_password in a seed migration |
gp / dental / pharmacy config/secrets/env.{dev,production}, .env |
DB_PASSWORD, CLIENT_DB_PASSWORD, REDIS_PASSWORD |
| Stale copies to purge | config/secrets/env.production.bak-20260923-1514, config/secrets/env.production.bak-refdata-20260926, .env.bak-refdata-20260926, and the pre-git tree dental_booking_app.pre-git-20260911-1145/ (its config/secrets/* and .env) |
Re-scanned 2026-10-13 by the app-engine side (I have no shell path there). The literal is present and
live: it is the requirepass on that host's local erpnext_redis and appears in
config/common_site_config.json (redis_cache / redis_queue / redis_socketio). Its local Postgres
is already clean — the .env POSTGRES_PASSWORD and the live root_password are not the literal (a
socket login made it look shared; a TCP test settled it). So the auxiliary host is redis-only work
with no database change, plus purging the stale erpnext-deploy.bak-20260912-150810/ tree. Rotating the
central credential alone would not break it, but leaving the literal there fails the "0 remaining hits"
evidence, so it is in scope.
The window is not empty. Jobs that fire inside it, and what to do with each:
| Time (Europe/London) | Job | Action for the window |
|---|---|---|
| 02:00 | backup-healthcheck.sh (every 2 h) |
leave — read-only freshness check; expect one alert if a backup is in flight |
| 02:30 | hermes-skills-sync.py (daily) |
quiesce — it writes the Wiki.js database, the very role whose password is changing |
| 03:00 | scan_trivy_weekly.sh (Sunday) |
quiesce — heavy scan, and its DefectDojo push would fail against a half-rotated stack |
| 03:30 | scan_nmap.sh ×2 (Sunday) |
quiesce — reads the Docker network mid-change and reports noise |
| 04:00 | docs-agent wiki housekeeping audit (Sunday) | quiesce — an AI workload on the same 4-core host, landing at the end of the window |
| every minute | /opt/restart-services.sh |
no longer quiesced — correction accepted 2026-10-13. It only restarts two website Node services (ports 3052/3053) and never touches DB-backed services, so it cannot mask a wrong-credential failure. My earlier claim that it could was wrong. |
| every 5–15 min | health, anomaly, egress, botnet, check-in, capacity collectors | leave running — this is the monitoring that proves the estate stayed healthy; the capacity collector reads /root/client-credentials*.json, which do not carry the shared literal today |
| every 2 min (app engine) | vps2-stack-guard.timer |
quiesce for the window — this is the real self-healer. It re-runs docker compose up -d --no-build for both booking stacks, so a container failing on a wrong credential would be silently brought back up and the failure would read as an intermittent restart loop instead of an error. |
| every 5 min (app engine) | container-watchdog |
quiesce for the window — same masking risk, and it alerts on container-count collapse while the stack is deliberately being changed. |
Sequence: disable the six jobs above before the backup step, run the rotation, complete verification
(step 11), re-enable them, and confirm each runs clean for one cycle. Where they are defined: four are
entries in root's crontab (hermes-skills-sync.py at 02:30, scan_trivy_weekly.sh at 03:00 on
Sunday and the docs-agent audit at 04:00 on Sunday) and two are in /etc/cron.d/security-scanner (the
two scan_nmap.sh lines at 03:30 on Sunday). Five therefore, not six — /opt/restart-services.sh was
removed from the list by the 2026-10-13 correction — plus two on the app engine:
vps2-stack-guard.timer (every 2 minutes) and the container-watchdog cron. Comment the lines out
rather than deleting them (units: systemctl stop + disable --now, restored the same way), so
re-enabling is exact. Record the disable/enable in the evidence section. Nothing else needs pausing, and
the rotation reboots no host.
preflight-backup.sh / post-restart-verify.sh on the primary do part of thisopenssl rand -base64 24..env value only matters at next initdb/recreate):ALTER ROLE postgres PASSWORD '<new>'; and, if decoupling, ALTER ROLE wikijs PASSWORD '<new>';fh_app_user owns nothing and has nofh_app_user is not a/opt/infra/.env (POSTGRES_PASSWORD, WIKI_DB_PASSWORD) so future recreations match.REDIS_PASSWORD in each app env, then restartredis + app / worker / beat on the app engine (requirepass changes need a Redis restart). On theerpnext_redis requirepass and theconfig/common_site_config.json — no database work there (2026-10-13 finding).config/secrets/env.production DB_PASSWORD, /etc/default/gp-booking-celery,/root/.config/veripath-db.conf, /opt/wagtail, /opt/cold-standby.db_password through Django (encrypted field — never raw SQL): a dataFIELD_ENCRYPTION_KEY..pyc, .bak-*, the July backup tree, the pre-git tree, theerpnext-deploy.bak-20260912-150810/) so the retired value is not left on any host,old-value once the window hasCHANNEL_LAYER_URL default should readREDIS_PASSWORD; the seed migration should not carry a plaintext db_password. (The six Hermespg_isready; Wiki.js loads; each app /infrastructure/health/; celery ping; aALTER ROLE … PASSWORD '<old>'; restart. Keep the old.bak-*, the July tree) hold the old value long after the rotation unless step 8The app-engine side has built the window's tooling under /root/rotation-20261018/: a carrier→role
mapping, a verifier, apply-rotation.sh, rollback-rotation.sh, generate-new-secrets.sh, a quiesce
script and the tenant-row check. It also holds old-value, a copy of the current credential kept for
rollback. The pack's finalize.sh purges it along with new-secrets.env and the local/remote backup
copies, and the verifier re-scans all three hosts (including /root and the pack directory itself) so the
closing evidence is an honest 0 hits rather than a claim. Its pre-window baseline is 85 hits, re-taken with the final scan roots on 2026-10-15 and
reproduced the same day by running the pack's own scan-literal.sh from the primary (45 primary + 26 app
engine + 8 auxiliary + container environments + the three role-auth probes). Nothing in the pack has been
executed. old-value sits inside the pack directory, which the scan deliberately excludes.
Scope correction for that evidence (2026-10-15) — and a correction of my own correction. I first recorded
that the pack's scan tooling carried "no reference to /etc/veripath/pgbouncer/" and was therefore blind to
it. That was wrong, and the way I got it wrong is worth keeping: I grepped the pack's scripts for the
pooler's path instead of reading which roots the scan covers. scan-literal.sh scans /etc on both the
primary and the app engine (plus /opt, /usr/local/bin, /root...), so it does cover the pooler's
directory and would flag the literal there had it been present. The pack is the app-engine side's artefact; the
correction and the carrier question were exchanged in from-hermes on
2026-10-15. Two further corrections went into the pack the same day, both recorded here as part of the evidence:
scan-literal.sh used to scan/root/rotation-20261018 — which holds old-value and pre-edit backup-* copies by design — so its ownfinalize.sh ran. That root is now excluded, and the exclusion isold-value; if thedental_booking_app/config/secrets/env.production.bak-wo-m3 holds the literal (it post-dates the 2026-10-13Adjacent change, deliberately outside the window: the app engine's overdue reboot is cleared on
2026-10-14 03:00 UTC by a one-off patch-bootstrap.timer (44 updates, then shutdown -r +1). Note the
opposite treatment of vps2-stack-guard.timer: it stays running for that reboot — after a reboot it
is what brings the booking stacks back — whereas the rotation window pauses it so a credential failure
cannot be masked and then look green.
After execution, record here: the window date/time, the decouple-vs-single-window choice, the
checklist count (45 primary occurrences + 23 app-engine + 8 auxiliary → 0 remaining hits), the
verification results from step 11, the jobs quiesced and re-enabled, the removal of the execution pack's
old-value, and the purge confirmation. That dated record, plus this runbook, is the evidence that
credential management is controlled rather than incidental.