¶ ⏭ RESUME HERE — next session (state as at 2026-09-11, end of Phase 2)
Phases 0–2 are executed and verified (§5). Two items remain — items 1 and 3 are
closed (below). Neither remaining item is blocked on engineering — each needs exactly one
external input: an OpenCode run, or your product decisions. Nothing else is outstanding.
# Remaining item What unblocks it Who 1 RefreshAGENTS.mdtopology sectionDONE 2026-09-11 — VPS1's line now records the dental retirement, plus two lines on the veripath-opsrepository and the Netdata collector intervalHermes 2 GP strand: commit the 36 items (incl. 4 untracked migrations), GP Redis healthcheck, ORTHANC token rotation an OpenCode run, while Hermes is stopped OpenCode 3 Pushveripath-opsto ForgejoDONE 2026-09-11 — the private repo matthew/veripath-opsnow exists andmasteris pushed at5a666cc(105 files, verified overgit ls-remoteand the Forgejo API). It was created through the Forgejo REST API with an existingwrite:repositorytoken authenticating asmatthew, because push-to-create is disabled on this instance.Hermes 4 Phases 3–5 (pharmacy + care home) product decisions, below you decide Do not re-discover these — they are settled: one running deployment per app; dental runs
only on the app engine (VPS2) from git, port 8010, migrations 0019–0031; VPS1 keeps the
dev clone but runs no dental runtime; the three original:8011vhosts are retired; the VPS2
stack guard (/opt/scripts/vps2-stack-guard.sh) now guards the new deployment; Orthanc stores
reconciled to 10 studies / 5 patients;client_vps_postgresis not a duplicate — leave it
running; images are at 55.4 GB after a 17.4 GB prune; GP images must be kept until item 2
(they are the only build of code that exists nowhere in git).Item 1 — applied 2026-09-11 (on approval). In
AGENTS.md§"3-VPS topology", VPS1's line
now records that the dental runtime is retired there (single home = VPS2; VPS1 keeps the
development source copy only), and two lines follow the topology list: ops config is
version-controlled at/root/work/veripath-ops, and the Netdata Docker/cgroup collector runs
at a 10 s interval (rate-limited — the fix for the VPS1 CPU burn).Item 4 — the decisions needed:
- Pharmacy and care home: new sectors inside the existing multi-tenant app (like
gp/dental/client), or separate applications (new codebases)? This is the whole
resource question — sectors cost two tenant rows, separate apps cost two stacks (~2 × the
VPS2 footprint we just measured). Ports 8020 / 8030 are already reserved for them.- Client addressing:
<client>.pharmacy.veripath.co.uk/<client>.carehome.veripath.co.uk?- Does each sector get its own database (as
veripath_dentaldoes), and do they reuse the
existing ERPNext instance for their accounts leg?Separate, already-waiting item (not part of this plan): the OpenCode inbox holds an OPEN
request for the UDA demo re-run and a one-page-at-a-time doc pass on the FP17 course-card /
billed-appointment copy. Parked 2026-09-11 on Matthew's instruction — do not start the demo
re-run or the doc pass until he unpauses it.
Status: PROPOSED — nothing has been executed. Written 2026-09-11 after a live
audit of all three VPSs. This page supersedes the premises of
VPS Re-Arrangement 1 and
VPS Re-Arrangement 2, which were written before
the estate was measured and contain assumptions that are no longer true (see §2).Goal: use the existing resources efficiently, keep the platform robust and agile,
and prepare for two new sector apps (pharmacy and care home) — without buying
another VPS unless the measurements say we must.Superseded in part, 2026-09-13: DefectDojo now runs on VPS1, not VPS3 (the 8090
forward ontun-aux-vps3was withdrawn with it) — see
VPS Re-Arrangement 4. Everything else on this page
stands, including the target three-node layout and the §4 rationale for putting app
runtimes where the CPU is spare. VPS3 is now ERPNext-only.
Audit method: read-only inspection of VPS1/VPS2/VPS3 (container inventory, resource
usage, port bindings, nginx routing, systemd units, cron jobs, compose files, tenant
and migration state), plus one reversible diagnostic experiment on VPS1 (Netdata
paused for ~20 s, then resumed — see §3, D1). No configuration was changed.
| VPS1 — gateway / data / AI | VPS2 — app engine | VPS3 — auxiliary | |
|---|---|---|---|
| vCPU | 4 | 4 | 1 |
| RAM | 15 GiB (≈11 GiB available) | 3.8 GiB (≈1.6 GiB available, 815 MiB swap in use) | 3.8 GiB (≈2.0 GiB available) |
| Disk | 232 GB, 52% used | 116 GB, 20% used | — |
| CPU load | load 4.8–6.2, ≈62% busy | load 0.14 (idle) | busy with ERPNext + scanning |
| Running containers | 25 (33 total) | 12 | 10 |
| Role | nginx gateway (22 vhosts), Keycloak SSO, central Postgres, client-DB Postgres, Wiki.js, MinIO, LiveKit, Forgejo, AES (4), patient portal, mail, security dashboard, Netdata | GP app (live, :8000), dental app (:8010), Celery workers/beat, Redis ×2, MailHog, Ollama, Orthanc | DefectDojo (5), ERPNext (4), one stray Redis |
Central Postgres logical databases: dsp_clinic (1.1 GB), wikijs (50 MB), keycloak
(17 MB), veripath_dental (11 MB), aes_portal. Client-VPS Postgres holds dsp_clinic_dev
(15 MB) and erpnext_test_client (42 MB).
Sectors and tenants in use: sectors gp, dental, client; tenants VeriPath GP
(dsp_clinic), VeriPath Dental (veripath_dental), Test Client (dsp_clinic_dev),
Test Client Dental (test_client_dental), Dev Client (mothballed 2026-08-16).
Stated growth target: no more than about half a dozen clients.
| Page 1 / 2 assumed | Measured reality |
|---|---|
| VPS1 is a mostly empty "hall" with 12 GiB spare; VPS2 is the bottleneck | VPS1's spare resource is RAM only — it is the CPU-constrained box (≈62% busy, and ≈1.8 of its 4 cores are being burned by the Docker/Netdata collection loop, §3 D1). VPS2 is CPU-idle (load 0.14) and disk-rich; its only constraint is RAM. |
| Redis is the memory hog; centralise it on VPS1 and free FPS2 memory | Measured Redis containers are 2–6 MiB each. Centralising Redis saves essentially nothing and adds a cross-host dependency for a per-stack cache. Decision (confirmed): one Redis per app stack. |
| Move the two Celery Beat schedulers to VPS1 | Celery Beat is 16–33 MiB at ~0% CPU. Moving it saves no memory and couples a scheduler to the gateway. Not adopted. |
| Adding two apps costs ~2.2 GiB ("≈1.1 GiB per vertical") | Measured per-vertical runtime footprint: GP ≈ 446 MiB (app 161 + worker 257 + beat 22 + redis 6), dental ≈ 563 MiB (app 138 + worker 387 + beat 33 + redis 5). Two new sectors ≈ 0.9–1.1 GiB, not 2.2 GiB. |
| Delivered as one Redis container under a central infrastructure compose file with new tunnel units | No such layout exists. The real conventions are: /opt/infra owns the hub network and core services; per-app compose projects live in app directories; tunnels are autossh systemd units; DB reachability for containers uses socat bridges; nginx always proxies to 127.0.0.1:<port>. |
| Nothing was said about VPS3, duplicate stacks, or boot-time bring-up | VPS3 is a 1 vCPU node running ERPNext and DefectDojo; there are duplicate live deployments of the dental app, duplicate GP companion containers, and the boot-time bring-up service fails in a 30-second restart loop. See §3. |
dockerd and containerd sit at ~86–95% CPU each, permanently, on a 4-core box. Pausing the
Netdata container for ~20 seconds dropped dockerd to 2.8% and containerd to 2.1%;
resuming returned both to ~95%. Netdata is collecting per-container cgroup/Docker metrics
for 25 containers at update_every = 1s, with no override config present in the container.
Measured effect: ≈1.8 of 4 cores (≈45% of the box) permanently lost, and the cause of the
"load average 5" that looks like application load.
Fix: give Netdata an explicit collector config (mounted from /opt/infra) raising
update_every for the Docker/cgroup collection to 5–15 s, or disable the Docker-API
collector entirely and keep cgroup-based metrics. Netdata's healthcheck briefly reported
unhealthy during the experiment and recovered on the next interval — expected, no impact.
Fixed 2026-09-11 (Phase 0.1): implemented as update_every: 10 on the docker_local
collector, set through a mounted config (/opt/infra/netdata → /etc/netdata) so it survives
container recreation. dockerd 86% → 9.6%, containerd 88% → 9.4%, VPS1 load 5.0 → 0.7.
infra-bringup.service restarts every 30 s (Restart=on-failure, RestartSec=30); each run
dies at the patient-portal step with a container-name conflict (the existing container carries
no compose project label, so compose refuses to create it), exits 1, and systemd retries —
10 failed runs in 5 minutes. Two consequences:
bring-up.sh does not manage VPS1's GP companion containers at all (Redis, MailHog andFixed 2026-09-11 (Phase 0.2): the portal step is now guarded and non-fatal, a new step 4b
starts the client-postgres companion service, and remote stacks are brought up through a
remote_compose helper that retries three times and never aborts the run. A full bring-up now
exits 0 with no warnings, reaching the VPS3 and sync steps for the first time.
The dental app is deployed twice, at two code vintages, writing to the same central
database (veripath_dental, currently migrated to 0031):
| VPS1 deployment | VPS2 deployment | |
|---|---|---|
| Code | git working copy, commit 99dbff27 (2026-09-10) |
plain copy, frozen ≈2026-09-08 14:35, no git |
| Migrations present in code | up to 0031 | up to 0027 |
| Serves | test-client-dental.dental.veripath.co.uk (and the patient-radiograph API path) |
dental.veripath.co.uk |
Both vhosts resolve to the same practice data. The newer code applied migrations 0028–0031 to
the shared database, so the deployment serving the main dental domain is running pre-redesign
code against a post-redesign schema — old code writing claims, charges and invoices into the
current billing model. The VPS1 routing is described in nginx as a temporary AI-radiography
demo override dated 2026-08-12 ("revert to the tunnel when the demo ends"); that temporary
override is now the live front door. This must be resolved before the estate is re-arranged.
Neither app directory on VPS2 is a git repository; the live GP and dental runtimes there are
plain copies. The only git working copies are on VPS1. Nothing prevents a runtime copy from
silently falling behind again — which is exactly how D3 arose. Fixing D3 without fixing the
deployment path just resets the clock.
docker inspect / docker events output. Replace with adevelopment/projects/compose-conventions has no entry for the dental app ports (8010/8011)
or the per-client DB tunnel ports (5435–5438), and no rule for "one running deployment per
app". Without that rule the duplicate-stack failure mode (D3, and the earlier duplicate
GP/dental containers on both hosts) will come back.
patient.veripath.co.uk was serving an expired certificate (expired 2026-09-01), so
browsers would refuse the patient portal. Root cause: it was the only certificate on the host
using certbot's standalone authenticator, which cannot bind port 80 while nginx is running
— so every automatic renewal failed silently and the cert ran out. Every other certificate
uses webroot or the nginx plugin and is valid. Fixed by adding the ACME http-01 challenge to
the vhost and re-issuing with webroot, so it now renews unattended like the rest; the domain
returns 200 and the certificate is valid to 2026-12-10. A dry-run renewal succeeds.
Two related items: the global renewal post-hook currently exits non-zero (so a future
renewal may not reload nginx automatically), and a stale duplicate demo.veripath.co.uk
renewal config (superseded by demo.veripath.co.uk-0001) was emitted as a certbot warning on
every run and has been moved aside.
Each host runs its own Orthanc container against its own volume, and until 2026-09-11 both
dental front doors were live: the primary VPS served test-client-dental… and the app engine
served dental.veripath.co.uk. Each therefore accumulated its own studies, and the datasets are
not equal:
| Files | Size | |
|---|---|---|
| Primary VPS (served test-client-dental, and the demo workspace) | 12 | 4.1 MB |
| App engine (served dental.veripath.co.uk; now the only runtime) | 3 | 668 KB |
Now that one deployment serves every dental vhost, the app reads a single Orthanc — so the
studies held only on the primary VPS are no longer visible in the app. This needs an explicit
decision, not an assumption: if that imagery is clinical or otherwise worth keeping it should
be reconciled into the surviving store through Orthanc's own API (archive export/import, so the
index stays consistent — copying files alone would not be indexed); if it is throwaway test
data it can be discarded. Until then the primary VPS's orthanc volume is left untouched, and it
is already covered by volume-backup.sh and premigration-snapshot.sh. Ollama is not affected:
both hosts carry the same embedding model.
Resolved 2026-09-11. Reconciled into the app-engine store using Orthanc's own API rather
than by copying files: nine studies were exported from the primary as study archives and
imported instance-by-instance via POST /instances (the zip-upload route returns 405 on this
build, so it was not usable). Result: 9 imported, 0 failed, and both stores now report
identical inventories — 10 studies / 5 patients each. The primary's volume was never modified
and its container is stopped rather than deleted, so the operation is reversible.
Worth recording for compliance peace of mind: the study descriptions in these archives identify
them as synthetic test imagery (the description text says so, with placeholder patient names),
so no real patient data was involved in the move. Patient identifiers were never printed.
Principles: one running deployment per app; runtimes together on the node with spare CPU;
gateway and data on the node that already holds them; every deployment built from git; every
port bound to loopback; nothing added to a node until that node's known defects are fixed.
VPS1 (gateway + data + dev source) VPS2 (application engine) VPS3 (auxiliary)
nginx + TLS (all vhosts) GP app engine DefectDojo
Keycloak SSO Dental app engine ERPNext (client accounting)
Central Postgres (+ client DB) Pharmacy app engine (new)
MinIO / LiveKit / Wiki.js / Forgejo Care-home app engine (new)
AES x4 / mail / portal / dashboard Celery workers + beat per app
Netdata (retuned, fast fix) one Redis per app stack
git working copies (dev source)
Decided 2026-09-11: dental's single home is VPS2, built from git; VPS1 keeps the
development source working copy only, not a runtime.
Rationale: VPS2 is the idle node (load 0.14 on 4 vCPU, 94 GB disk free) — application
runtimes belong where there is spare CPU, and its RAM gap is answered by removing the
duplicate containers, not by relocating caches. VPS1 keeps what it is already good at
(routing, identity, data, object storage, source) once its CPU burn is fixed. This is the
inverse of page 2's conclusion, and it is driven by measurement rather than metaphor.
VPS4: not now. At ≤6 clients, the audit found the equivalent of ~1.8 CPU cores and ~60 GB
of disk already being wasted on VPS1 — recover that first. Revisit a 4th node only if, after
Phase 1, either box sustains >70% CPU or >85% RAM, or the pharmacy/care-home stacks prove
materially heavier than the GP/dental pattern. If a single upgrade is needed first, the
cheapest lever is VPS3's 1 vCPU (it must absorb scanning and ERPNext work for four
codebases), followed by VPS2 RAM (vertical, cheap) — both cheaper and less disruptive
than a new node.
Each phase ends with a verification step; nothing proceeds on an unverified assumption.
| # | Task | Owner | Acceptance check |
|---|---|---|---|
| 0.1 | Retune Netdata container collection (D1) | OpenCode | dockerd/containerd < 10% idle; VPS1 load < 2 |
| 0.2 | Make bring-up idempotent and complete (D2): fix the patient-portal name conflict; include the VPS1 GP companion containers or move them with the app | OpenCode | systemctl status infra-bringup = active (exited 0); full run reaches VPS3 steps |
| 0.3 | Reclaim disk: prune unused images, build cache, stopped containers (D5) | OpenCode | docker system df reclaimable < 10 GB |
| 0.4 | Secret hygiene: Redis password out of the healthcheck command line; webhook token out of plaintext config (D5) | OpenCode | docker inspect shows no credential; token rotated |
| 0.5 | Record before/after metrics on this page | Hermes | numbers appended below |
| 0.6 | Add the missing port/allocation entries and the "one running deployment per app" rule to compose-conventions (D6) |
Hermes | page updated |
Phase 0 is zero-risk-to-users: no routing, data or schema changes, and it recovers CPU
and disk that is currently being wasted.
| # | Task | Result |
|---|---|---|
| 0.1 | Netdata retune | Done. update_every: 10 via a mounted config. dockerd 86% → 9.6%, containerd 88% → 9.4%, load 5.0 → 0.7: ~1.8 cores returned to the host. |
| 0.2 | Bring-up fix | Done. Portal step guarded and non-fatal, client-postgres now started, remote stacks retried. Full run exits 0 with no warnings and reaches every step; the systemd unit settled to active (exited) and the 30-second restart loop has stopped. |
| 0.3 | Disk reclaim | Partial. Build cache pruned (6.07 GB reclaimed). Unused-image pruning deferred deliberately: some images are currently the only build of code that is not yet in git, so the full prune waits until that work is committed. |
| 0.4 | Secret hygiene | Partial. The Redis password no longer sits on a healthcheck command line in the dental compose (committed ac7b30e3, validated on a throwaway container; applies at the next recreate). GP's equivalent and the ORTHANC webhook token rotation are handed to OpenCode. |
| 0.5 | Record metrics | Done (this table). |
| 0.6 | Conventions update | Open — port table plus the "one running deployment per app" rule. |
Before any edit, a snapshot of the unversioned ops config was taken:
/root/backups/ops-config-20260911-1124.tar.gz (compose files, scripts, tunnel and bring-up
units, nginx sites). Putting those paths under version control is still outstanding.
Steps 1–2 executed 2026-09-11 and verified. While the audit was being written up, the
main dental domain was found serving 502, so the cutover was done immediately rather than
as a scheduled step. The precise cause is worth recording, because it is a textbook case of
the drift this page is about:
The app-engine copy had been made from a compose file written for the primary VPS: it
hardcodes DB_HOST=gp_booking_postgres (a name that resolves only on the primary hub
network) and publishes the app on 8011, while the tunnel between the hosts meets 8010.
The running container still carried the older port mapping, so the domain worked by
accident until that container was recreated.
Fixing the bring-up loop (Phase 0.2) allowed the app-engine step to run for the first time,
which recreated that container and applied the drift: nothing then listened on 8010, and the
app could not resolve the database host name, so it sat at "Waiting for PostgreSQL…". The
domain went 502 as a result.
So the cutover was not merely tidying: it removed a deployment that could not work on that
host as configured, and whose code also predates migrations 0028–0031. Both defects now have
a permanent fix rather than a patch: the compose takes DB_HOST, DB_PORT and APP_PORT
from the stack .env (commit 6e75c5ac), so one file deploys correctly to either host, and
the runtime is rebuilt from git instead of hand-copied (6e75c5ac deploy on the app engine).
Both dental vhosts now point at the current build on VPS1 (dental.veripath.co.uk moved from
the tunnel port to the local runtime port; the test-client vhost already used it). The
current build resolves tenants correctly for both hostnames (verified with a Host-header
probe before the switch). dental.veripath.co.uk returns 200.
VPS2's stale app, worker and beat containers are stopped, not deleted (rollback: revert
the one nginx line and start them again).
Guard correction: VPS2's vps2-stack-guard.sh lists those containers as required and
would have silently restarted the stale stack within two minutes. Its requirement list was
reduced to dental_orthanc (still running, serves DICOM), with a header explaining why and
how to restore it once VPS2 runs a git-based dental deployment. Verified: the containers stay
stopped across a guard cycle and service is unaffected. Backup of the previous guard:
/opt/scripts/vps2-stack-guard.sh.bak-20260911 (on VPS2).
Remaining (Phase 1 steps 3–4): rebuild dental on VPS2 from git, verify, move both vhosts
there, retire the VPS1 runtime; then retire the duplicate companion containers.
Step 3 executed and verified 2026-09-11. The app engine now runs the single dental runtime,
deployed from git rather than hand-copied:
DB_HOST/DB_PORT/APP_PORT from the stack .env (commit 6e75c5ac), so the same filefeature/dental-nhs-accounting at 6e75c5ac, with the non-gitmedia/, staticfiles/, data/, opencode_tasks/); thedental_booking_app.pre-git-<timestamp> for rollback.0031 applied, Celery worker ready, and both dental hostnamesdental.veripath.co.uk,test-client-dental.dental.veripath.co.uk, and the /api/ path ontest-client-dental.patient.veripath.co.uk. The retirement of the primary-VPS runtime wasStep 4 executed 2026-09-11 (D8 resolved first). Pre-flight verified that no running
container, cron job or script on the primary host referenced the duplicates, and that their
ports were not published. Then retired on the primary (stopped, not deleted, volumes intact):
gp_booking_redis, mailhog, dental_redis, dental_ollama, dental_orthanc. The client
tenant database container was deliberately left running and is healthy — it is the DB for
tunnelled client tenants and is started by bring-up step 4b. All seven public endpoints stayed
200 through the change.
Decision: dental's single home is VPS2, built from git; VPS1 keeps the development
working copy but no longer runs a dental runtime. This fix is applied as its own small,
reversible change ahead of the wider re-arrangement, because a stale deployment writing into
a schema migrated by newer code is a data-integrity issue, not a tidiness one — claims,
charges and invoices are being written by pre-redesign code against the current billing model.
Gate — passed 2026-09-11. The current build is not demo-configured: VPS1's uncommitted
local changes are a branding context processor (one settings line, one new file, one template
line) with no divergence in ALLOWED_HOSTS, CSRF, TLS-header or SSO settings between the two
deployments, and both running containers allow-list both dental hostnames. Before cutover,
still confirm per-vhost tenant resolution and take a database snapshot.
dental.veripath.co.ukproxy_pass change from the tunnel port to the local runtimedocker compose stop — never down,Acceptance: one dental runtime serving both vhosts, code and schema both at 0031, and no
pre-redesign writer anywhere in the estate.
compose build),bring-up.sh, the tunnel units and the status/health scripts so they reflect whereNot yet decided product-wise (research in progress: routing-based sectors inside the existing
codebase vs separate Django apps). The infrastructure work is the same either way and follows
the existing SIAAS pattern:
{slug}_pharmacy / {slug}_carehome style, following the GP/dental{slug}.<sector>.veripath.co.uk), proxying to 127.0.0.1:<port>, with the same imagerysocat bridge unit on VPS1, matching permitlisten entries on the deploy key, and theSizing: two new stacks ≈ 0.9–1.1 GiB RAM and a small CPU burst. With the duplicates removed
and caps applied that fits VPS2 alongside GP and dental; if measurements show otherwise, VPS2
RAM is the cheap vertical upgrade, before considering a new node.
Rehearse end-to-end with one client VPS: provision, tunnel, tenant row, onboarding, backup,
restore drill, kill-switch test.
Update this page, add "premises superseded" notes to pages 1 and 2, refresh AGENTS.md and
the operations guide, and add the new sectors to the port table.
Phase 0 and Phase 1 are complete and verified (results tables above). What remains in this
plan:
/opt/infra, /opt/scripts, the bring-up/tunnel units, the/root/work/veripath-ops (105 files, baseline commit 5a666cc), with credentials excluded by.env/key files are stripped on collection, and the only matches forhermes verify passes for that project again (bootstrap and buildAGENTS.md — still to do, blocked on consent. Its topology section is now partly staleSectorDatabaseRouter / tenant model) or{slug}.<sector>.veripath.co.ukmatthew/dental_booking_appgit.veripath.co.uk vhost with no IPnhs_private_key.pem (the NHS/EPSclinical_data/integrations/nhs_auth.py) and a dev TLS key — was anonymously2026-09-11 (6) in agents/handoff/from-hermes). The same key is byte-identical ingp_booking_app (private, never anonymously served) — one shared key, so one rotation covers both.matthew/veripath-ops created as a private repository andmaster = 5a666cc, 105 files / 68 KB, confirmed overgit ls-remote and the Forgejo API (private: true, empty: false, default branch master).write:repository token authenticating as matthew; the push itself used the.env/key file is tracked and aAGENTS.md topology section refreshed on approval: VPS1veripath-ops repository and Netdata's 10 sveripath-ops======… / Page: … artifact left at the foot of this page by an earlier paste-back--get output was removed in the same write.hermes verify passing again (ce2e1a9e); 17.4 GB of unused images reclaimed with GP buildsAGENTS.md remains blocked pending explicit consent./root/work/veripath-ops (previously no history and no rollback for the files this work.env, one runtime per app, git-based deploys, no secrets on6e75c5ac with the compose made host-portable; all three consumers cut over and verified;dental.veripath.co.uk was foundpatient.veripath.co.ukac7b30e3); tenant-brand commit pushed (d39def75);