Status: Resolved
Severity: Medium (isolated to test-client tenant; main GP site unaffected)
Detected: 2026-08-01 02:14 UTC (after scheduled monthly reboot)
Resolved: 2026-08-01 ~16:40 UTC (primary); follow-up stale DB-name issue fixed 18:00 UTC
Affected:https://test-client.gp.veripath.co.uk/returnedServer Error (500); the Infrastructure dashboard showedVPN tunnel unhealthy: db_query: ... port 5435 failed: server closed the connection unexpectedly
Related: test-client login routing incident
After the host's scheduled monthly reboot (02:13 UTC on the 1st of the month), the test-client tenant could not reach its database. Every request to test-client.gp.veripath.co.uk produced a Django 500 (django.db.utils.OperationalError: connection to server at "172.18.0.1", port 5435 failed: server closed the connection unexpectedly), and the Infrastructure dashboard reported the client VPN tunnel as unhealthy. The main GP site (gp.veripath.co.uk) was unaffected.
The test-client tenant is routed, via the multi-tenant router, to its own isolated PostgreSQL database (dsp_clinic_dev) inside the client_vps_postgres container. The connection path is:
gp_booking_app (Django)
-> 172.18.0.1:5435 (host socat forwarder: erpnext-tunnel-forward.service)
-> 127.0.0.1:5435 (docker-proxy host port binding)
-> client_vps_postgres:5432
Two independent problems combined:
The client_vps_postgres container was not publishing port 5435 to the host. The host socat at 172.18.0.1:5435 forwards to 127.0.0.1:5435, but nothing was listening there (the docker-proxy binding for port 5435 did not exist). Connections were accepted by socat and then immediately closed, producing server closed the connection unexpectedly.
client_vps_postgres container was misconfiguredThe container mounted the same named volume as the main Postgres (gp_booking_app_postgres_data). Its image (postgres:15-alpine, postgres UID 70) could not read data files owned by the main image's postgres UID (999), so every query failed with Permission denied on global/pg_filenode.map. Even if the tunnel had been intact, the container could not serve the database.
The repo's migration history contained conflicts (duplicate/renamed migrations, e.g. users/0011b vs 0011, appointments/0023 and 0027 both adding staff_role_required). The main DB had absorbed these historically; a fresh client DB could not replay them. This is why a clean schema rebuild was needed rather than migrate.
| Time (UTC) | Event |
|---|---|
| 02:13 | Host rebooted (scheduled monthly reboot — recurs on the 1st of each month) |
| 02:14 | Containers restarted; socat forwarders came back up, but 127.0.0.1:5435 had no listener |
| 02:15+ | Tunnel health checks began failing; test-client returned 500 |
| 15:20 | Incident investigated; tenant DB config found pointing at 172.18.0.1:5435 |
| 15:58 | client_vps_postgres recreated with dedicated volume + published 127.0.0.1:5435:5432 |
| 16:05 | dsp_clinic_dev restored from pre-migration backup (/root/backups/pre-migration/client_vps_2026-07-22.sql.gz) |
| 16:38 | Client DB schema rebuilt from main DB schema (migration graph was inconsistent) |
| 16:42 | gp_booking_app restarted; test-client began serving (redirect to Keycloak login) |
| 16:45 | Tunnel health monitor fixed and verified healthy |
docker-compose.yml for client-postgres:
client_postgres_data (no longer shares the main Postgres volume) — resolves the UID 70/999 permission conflict.127.0.0.1:5435:5432 so the host socat chain reaches the container (restores the missing hop)../postgres/... instead of ../work/gp_booking_app/postgres/...).client_vps_postgres (healthy) and set the fh_app_user password to the value stored in the tenant config.dsp_clinic_dev: the Jul 22 backup contained no meaningful data (0 patients/appointments) and an inconsistent migration history, so the schema was cloned from the working main DB (118 tables, 198 migrations recorded, 0 unapplied). Client DB is schema-only; test data must be seeded.gp_booking_app, gp_booking_celery_worker, gp_booking_celery_beat.client_monitor/tasks.py): it was connecting to tunnel_ip:5432 (host Postgres — the wrong endpoint) and writing tenant=None into a non-nullable FK. It now resolves the tenant and uses the tenant's DB endpoint.test-client.gp.veripath.co.uk → 302 → Keycloak test-client realm login; main GP site still 200; tunnel health monitor reports healthy=True.After the primary fix, the Infrastructure dashboard intermittently showed VPN tunnel unhealthy: db_query: psql: error: ... FATAL: database "practice_app_db" does not exist every 15 minutes, even though the tenant DB (dsp_clinic_dev) was healthy.
ClientTunnelHealth table. The fixed Celery task (check_all_tunnel_health) uses the tenant config and wrote healthy=True, but the external cron script /opt/check-tunnel-health.py (via /etc/cron.d/tunnel-health, every 15 min) read its credentials from /root/client-credentials.json, which still contained the old database name practice_app_db. It then wrote a healthy=False record that overwrote the good one, so the dashboard showed a stale failure./root/client-credentials.json → "database": { "name": "dsp_clinic_dev", ... } (password/user already matched the tenant config). Verified /opt/check-tunnel-health.py --verbose → RESULT: TUNNEL HEALTHY and the latest ClientTunnelHealth row is overall_healthy=True.practice_app_db references remain in compose, postgres config, or systemd.# All containers up?
docker ps --filter name=client_vps --filter name=gp_booking
# Tunnel chain listener present?
ss -tlnp | grep -E ':5435'
# Tunnel health monitor (should be healthy)
docker exec gp_booking_app python -c "from client_monitor.tasks import check_all_tunnel_health; check_all_tunnel_health()"
docker compose up after a reboot, not just relying on restart: unless-stopped. The compose file now correctly defines the port publishing and volume, so the chain is restored consistently.client_postgres_data).tenancy_tenant) points at db_host=172.18.0.1, db_port=5435; the compose must publish 127.0.0.1:5435:5432 on client-postgres. If either changes, both must change.migrate instead of schema cloning. Until then, use the schema-clone procedure documented in this page.ClientTunnelHealth. Consider alerting on overall_healthy=False so incidents are caught before users report them.python manage.py seed_patients (or scripts/seed_patients.py).Incidents/2026-08-01-test-client-login-routing) was found during validation of this one: the tenant admin landed on the VeriPath staff dashboard. Verify a client-tenant ADMIN login lands on /partner/, not /dashboard/admin/./root/client-credentials.json in sync with the tenant config. The external cron health check (/opt/check-tunnel-health.py via /etc/cron.d/tunnel-health) reads the DB name/user/password from this file. If the tenant DB is renamed or recreated, update both the tenant (tenancy_tenant) AND this file — otherwise the cron script overwrites good ClientTunnelHealth records with stale failures.tenancy/routers.pyerpnext-tunnel-forward.service (172.18.0.1:5435 → 127.0.0.1:5435), client-db-forwarder.service (172.18.0.1:15432 → 127.0.0.1:15432)client_monitor/tasks.py/root/backups/pre-migration/client_vps_2026-07-22.sql.gz/root/work/gp_booking_app (branch main, HEAD fecacb1)