Plan based on the research in
Primary VPS Restructuring.
Goal: stop the AI workload (nativehermesprocesses + opencode'sesocket5,
observed at 30–50% CPU each) from taking down the production booking apps
(especially the dental app) by physically separating the AI/dev tooling from
the booking applications across three VPS instances.
VPS2 is restored as a primary booking-apps host. The 3-VPS restore
(Option A) is DONE and verified by OpenCode (see
agents/handoff/from-opencode, note
2026-08-16). This section supersedes any earlier "outage" status.
docker compose down (bothdocker system prune -a --volumes --force → docker compose pull → docker compose up -d. The prune also deleted the two shared externalgp_booking_app_gp_booking_network,aes-service_aes-network), so the recovery up -d failed and VPS2 was leftgp_booking_app :8000,dental_booking_app :8010, plus celery worker/beat, redis, orthanc) running212.227.199.114); reached from VPS1 viatun-apps-vps2.service; VPS2 reaches VPS1 DBs (central Postgres172.18.0.1:5433, client DB 5435, tenant DBs 5436/5437, AES 8004) viatun-vps1.services.service. VPS3 (DefectDojo :8090, ERPNext :8005) also backtun-aux-vps3.service.vps2-stack-guard.service + .timer (every 2 min) — recreates the externaldocker compose up -d --no-build for both stacks if any/etc/profile.d/vps2-guard.sh — blocks docker compose down anddocker system prune in VPS2 shells (loud warning; docker compose stopcontainer-watchdog.sh (5-min cron) — alerts via the VPS1 mailer if the/opt/infra on VPS1 — hub network):docker compose down, docker system prune -a --volumes --force. Usedocker compose stop.Everything below in the original plan stands, with these verified corrections /
findings from pre-work:
| Box | Role | Specs (verified) |
|---|---|---|
88.208.212.211 (this host) |
VPS 1 — Existing Primary (gateway + data + AI/dev) | 16GB / 232GB / 4 CPU |
212.227.199.114 |
VPS 2 — Core Booking Apps | 4GB / 120G / 4 CPU (AMD EPYC-Milan), no Docker yet |
77.68.112.126 |
VPS 3 — Auxiliary (DefectDojo + ERPNext) | 4GB / 120G disk but root partition only 58.9G / 1 CPU |
⚠️ VPS 3 (
77.68.112.126) is 1 CPU, not the 4 CPU the plan assumed. It is
suitable for the on-demand-only auxiliary role but is NOT a good booking-app
host. VPS 2 is the booking-apps box. Also: its 120G disk has only a 58.9G root
partition — it was previously used as a backup target; that backup role is
retired and the box can be reset (grow partition to the full 120G).
The original assumption blamed "native hermes + esocket5". Verified reality:
local_hermes_agent container crash-loops (image nousresearch/hermes-agent:latest).dashboard, main-hermes, gateway-default) andRestartCount reached 4096. Everypkill -9 -f hermes killed the container's main process → Docker restarted thedashboard service (the hermes dashboard --port 9119 process) at nice 0, pegging ~1 full core. Vicious cycle.
docker stop local_hermes_agent (policy unless-stopped → staysdocker start local_hermes_agent.celery-worker.service, celery-beat.service)/var/www/html/gp_booking_app venv. After a pkill -f celerygp_booking_celery_worker etc.) and was unaffected.CPUQuota=150% on hermes-demo.service" is/opt/scripts/cpu-guard.sh (1-min cron) stopslocal_hermes_agent if it ever starts and renicies any hermes/esocket/opencode process to nice 19. Remove after migration.77.68.112.126 was used as a backup target; that role is no longer/root/cleanup-quarantine-20260811/): backup-client.py, restore-client.py,/etc/practice-backup.conf, /etc/cron.d/practice-backup, /etc/cron.d/backup-health,/usr/local/bin/backup-healthcheck.sh, /var/backups/practice/./root/clientadmin-key +/root/client-credentials.json (used by check-tunnel-health.py andcollect-capacity.py)./opt/scripts/primary-backup.sh,volume-backup.sh, /var/backups/daily) and the pre-migration snapshot tool/opt/scripts/premigration-snapshot.sh.wg0/51820 aretun-postgres.service forwards 127.0.0.1:5433 → VPS1 Postgrespsql query of dsp_clinic).tun-apps-vps2.service forwards 127.0.0.1:8000/:8010 → VPS2 appstun-aux-vps3.service forwards 127.0.0.1:8080 → VPS3 DefectDojo (active).192.168.100.x addressing is dropped. nginx and app configs127.0.0.1:<port> instead. Postgres stays bound to 127.0.0.1:5433 (no rebindDB_HOST=127.0.0.1 on VPS2./tmp/bobs asatds/atdd), not just the AI workload. It respawned via the container's s6 andhermes/celery/esocket5.esocket5 was malware (socket scanner in the attacker's toolkit at/tmp/.idcc/ds/), not AI tooling. The plan's original "opencode esocket5" framing/tmp artifacts removed, Forgejoesocket* processes in addition to renicing AI processes.212.227.199.114, 4GB/120G/4CPU). Verified live:gp.veripath.co.uk and dental.veripath.co.uk both serve HTTP 200/302 from VPS2/infrastructure/health/ returns {"status":"ok"};172.18.0.1:5433 (Docker bridge gateway), NOT127.0.0.1 — app containers can't reach the host's loopback, but can reach theDB_HOST=172.18.0.1, DB_PORT=5433 in bothenv.production files. AES portal tunnel on 172.18.0.1:8004.AES_API_BASE_URL and LIVEKIT_URL now read from env in both settings.pyAES_API_BASE_URL=http://172.18.0.1:8004.--concurrency=2 in both compose files./static/ from local dirs/var/www/gp_booking_app_static/, /var/www/dental_booking_app_static/)/opt/scripts/sync-static.sh) — the apps do not/static/ to them 404s.auto_no_show cron now runs against VPS2's app over SSH.docker compose down'd (volumes kept as backup). VPS1 retainsgp_booking_postgres (central DB) + client_db_dental_dev (deferred ERPNext).settings.py changes committed to Forgejo (see cleanuptenancy point at the local Docker172.18.0.1:<port>). After the move:
test-client (5435) → client_vps_postgres is on VPS2 → its compose port was127.0.0.1:5435 to 172.18.0.1:5435.test-client-dental (5436) and dev-client (5437) → DBs stayed on VPS1 →172.18.0.1:5436/5437.dsp_clinic_dev,test_client_dental).aes.veripath.co.uk failed with unauthorized_client ataes-portal container had an emptyCLIENT_SECRET (compose ${AES_PORTAL_CLIENT_SECRET:-} had no .env). Created/opt/aes-service/.env with the correct Keycloak secret and recreated the172.18.0.1:8004) and the aes-signer-sa service account token exchange both OK.check-tunnel-health.py check for the172.18.0.1:5435) now reaches VPS2's client_vps_postgres via aerpnext-tunnel-forward socat that77.68.97.101 uptime monitoring has been failing since early Julysettings.py committed + pushed to Forgejo in both reposgp_booking_app main → a345980, dental_booking_app develop → 7b1b3235):172.18.0.1:5436/5437/15432):client-db-forwarder, dental-tunnel-forward — each ~30,500 restarts, "Addresstenant-gateway@.service template. Manual socats were killed,tenant-gateway@15432/5436/5437 instances nowerpnext-tunnel-forward.servicetenant-gateway@5435.service disabled — it could never bind because thetun-apps-vps2.service) already owns 172.18.0.1:5435; thattun-postgres.service removed from VPS2 — it was failed/disabled andtun-vps1.services.service (which forwards centralenv.production (both apps) synced to the VPS2 values — DB_HOST=172.18.0.1,DB_PORT=5433, plus AES_API_BASE_URL=http://172.18.0.1:8004, so the canonical repo/opt/scripts/cpu-guard.sh, 1-min cron) — theesocket* malware-familyveripath-webhook.service (runs/opt/deploy-webhook.js on 127.0.0.1:3051) had accumulated 88,581 restarts —node /opt/deploy-webhook.js (pid 744, up since Aug 01) held port 3051 soEADDRINUSE). Manual process killed; the systemdNRestarts=0, active). Verified end-to-end through nginxgit.veripath.co.uk/webhooks/deploy): unsigned POST → 403 (HMAC guard working),veripath-site, URLhttp://git.veripath.co.uk/webhooks/deploy) is unchanged and works.server/webhook-server.mjs (port 3054) was arestart-services.sh watcher and the deploy script. Removed the 3054/opt/restart-services.sh and /opt/deploy-veripath-site.sh, and/dashboard/compliance-admin/ fixed. After Keycloak login,SYSTEM_ADMIN → 403. Fix (commit a696e57, pushed to Forgejo):
ComplianceAdminDashboardView, AgentIntegrationView, LynisSecurityScanView,LynisScanDetailView now allow ['SYSTEM_ADMIN', 'ADMIN']PlatformAdminRequiredMixin (core/permissions.py) and SuperuserRequiredMixin['SYSTEM_ADMIN', 'ADMIN']PARTNER_ADMIN etc.) remain excluded; ADMIN = VeriPath staff,PARTNER_ADMIN = client.matthew (ADMIN) account: compliance-admin, system-iar,gp + dental health endpoints {"status":"ok"} through thecheck-tunnel-health.py TUNNEL HEALTHY, no systemd crash-loops remain.Everything that is not a production booking app, plus the AI/dev tooling that
caused the outages:
gp_booking_postgres — stays here (decision made to avoidesocket5 — the AI load, now harmlesslylocal_hermes_agentgp_booking_app stack: app, gp_booking_celery_worker, gp_booking_celery_beat,gp_booking_redis, client_vps_postgres, mailhogdental_booking_app stack: dental_app, dental_celery_worker,dental_celery_beat, dental_redis, dental_orthanc⚠️ VPS 3 is actually 1 CPU (see Pre-work status) — keep DefectDojo/ERPNext strictly
on-demand and consider whether ERPNext should be deferred entirely.
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
free -h
127.0.0.1:5433 (central Postgres) — verified working.127.0.0.1:8000/:8010 (apps) — enable in Phase 3.127.0.0.1:8080 (DefectDojo) — active.192.168.100.x subnet; use 127.0.0.1:<port> everywhere./opt/defectdojo configuration to VPS 3.defectdojo_defectdojo_postgres volume (2.0G) and media volumes to/root/work/erpnext-deploy and the erpnext_sites_data volume (5.3M) todocker compose down on/root/work/gp_booking_app and /root/work/dental_booking_appstaticfiles and media to VPS 2.docker compose up, otherwisemkdir -p /var/lynis-agent /vaults/sidebar-notes /usr/local/bin/lynis-agent
lynis-agent binary from VPS 1 and populate the vault notes./usr/local/lynis and /usr/local/bin/lynis bind-mounts arelynis is already apt-get install'd inside the image, so thoseconfig/secrets/env.production so DB_HOST=127.0.0.1 (VPS 2 reaches VPS 1'stun-postgres.service SSH tunnel on port 5433). No rebind of192.168.100.1:5433127.0.0.1:5433 on VPS 1; the SSH tunnel is the only path. It is never bound to0.0.0.0, so it stays off the public internet.ServerAliveInterval=30 already set in the tunnel units).celery -A dsp_clinic worker --concurrency=2 --loglevel=info (currentlydeploy.limits in both compose files to measured real usage.docker compose down on VPS 1.gp.veripath.co.uk → http://127.0.0.1:8000 (SSH tunnel to VPS 2)dental.veripath.co.uk → http://127.0.0.1:8010 (SSH tunnel to VPS 2)http://127.0.0.1:8080 (SSH tunnel to VPS 3)tun-apps-vps2.service on VPS 1 before this step so the ports are bound.)location /static/ blockhttp://127.0.0.1:8000 over the SSH tunnel (no synccollectstatic into the bind-mounted./staticfiles/ on VPS 2 at every start) — not the stale local/var/www/gp_booking_app_static/.nginx -t then systemctl reload nginx.--concurrency=2 in both apps — currently unbounded), tighten deploy.limits towiki_content volume is/var/lynis-agent, /vaults/sidebar-notes and /usr/local/lynis from the host.mkdir -p and populate /var/lynis-agent, /vaults/sidebar-notes/usr/local/bin/lynis-agent on VPS 2 before docker compose up. lynisapt-get install'd inside the image, so the /usr/local/lynis and/usr/local/bin/lynis bind-mounts are likely redundant — check and drop them.gp.veripath.co.uk static file serving. Nginx serves /static/ from a local/var/www/gp_booking_app_static/ alias. Mitigation: add a location /static/http://192.168.100.2:8000 over the tunnel — thecollectstatic into the bind-mounted ./staticfiles/ on VPS 2 at