Status: LIVE DISCUSSION PAGE, opened 2026-10-13 at Matthew's request — continues 2026-10-14. Scaling is its
own topic, closely aligned to patching but not the same question: maintenance is what exposed it, but the
estate has to scale whether or not we ever run another patch.
Related: Client VPS Patching — Managed Service (the maintenance
design, §9 and §10), Platform Capacity & Growth Plan (triggers and
purchase order), Client VPS (the SIAAS model),
Understanding Your Infrastructure Dashboard (what a client
sees), Disk Capacity Review — VPS1, and
The Scalability Rig — How to Use It, and How to Read What It Says
(the instrument, and how to read what it says).
Patching forces us to touch every client box on a schedule, which is what makes scale urgent. But the
estate has to scale regardless of maintenance: client databases, the application traffic that serves every
practice from our tier, the database access path, monitoring and collection, identity, backups, and the
maintenance plane itself.
So the patch requirement raised the question; the answers are estate-wide, not maintenance-specific. The
maintenance detail lives in the patching spec (§9, §10) and the trigger mechanics live in the capacity plan.
This page holds the reasoning and the decisions.
We cannot demonstrate that today's three-VPS tier holds at 200–300 clients, so assume it does not hold
until measured — and pre-plan the signs that tell us when to act. A contingency with a trigger is as good
as a solution.
Four working rules fall out of that:
| # | Lever | What it is | State (2026-10-13) |
|---|---|---|---|
| 1 | Transport | Today every client adds a listener, a systemd unit, an sshd session, a published port and host keys to the primary — and puts the primary in the data path of every client's database. The answer is a private overlay (WireGuard) with routed addressing. | First thing to break · Open, mechanism blocked 2026-10-14 — see the note below the table. A project in its own right (4–8 weeks). Trigger: >10 client boxes, or hub bandwidth >40% of link |
| 2 | Control plane | Agent built into the provisioning pipeline, poll-for-work (no inbound access), and wave rollout behind a health gate so one bad patch cannot reach every practice in a night | Agreed in principle · build briefs WO-A…D await rulings |
| 3 | Access model | The build visit is the only moment we hold access; after handover the box works and reports outbound, and disruptive steps can wait for the client's own admin to authorise | Agreed in principle · recovery path still open |
| 4 | Data volume | 15-minute polling is the row driver: ~192 snapshots/day for 2 boxes → ~10.5M rows/year at 300. Retention + rollup is cheap now, harder later | Not started · cheap and should precede growth |
| 5 | Platform tier capacity | The shared resources: central Postgres, the app tier (a 3.8 GB box), the tunnel hub, collection. Grows with the whole estate, not one client | Baseline measured; triggers + purchase order written (capacity plan) |
| 6 | Evidence & isolation | At hundreds of clients, per-tenant scoping stops being a nicety and becomes a gate; client packs must be generated automatically | Open · the tenant-scoping leak on the client dashboard is the first item |
| 7 | Onboarding at scale | The build visit must install and verify the agent, backup and reporting before the temporary credential is surrendered | Open · the build-visit runbook is a deliverable |
Measured 2026-10-14 — detail on the apparatus and the maths
and the reachability gate:
- Lever 1's mechanism is blocked upstream of the host. Inbound UDP to the primary never reaches it from
two unrelated networks, while TCP to the same address answers in milliseconds. A WireGuard overlay over
UDP therefore cannot be deployed as drawn; the project needs UDP-over-TCP wrapping, a different hub
provider, or a different transport. The overlay already standing on the primary and the app engine
(two interfaces, peers configured, keepalives on) has never completed a handshake — 635 KiB sent,
0 bytes received, which is the same block seen from the inside.- Lever 4's row volume is larger than stated. 96 capacity rows plus ~201 tunnel-health rows per
client per day (~297 rows, ~40 kB) — so at 300 clients the capacity table reaches the 5M-row trigger
56 days after that count is reached, not in the distant future.- The collection cliff is closed. One cron line per client, all firing in the same minute, each opening
its own connection to the central instance, was a hard stop at ~91 clients againstmax_connections
= 100. Replaced 2026-10-14 by a single staggered sweep (one cron line for any number of clients);
deployed and verified.- The app tier is what binds — but the first diagnosis of it was wrong. Measured properly on
2026-10-14: synchronous views do not serialise (Django gives each request its own thread), so
worker processes do not move the ceiling — throughput plateaus at 25–30 req/s for a tenant page and
is flat between 10 and 40 concurrent clients. What binds is two fresh database connections per
request (~50 ms each over the tunnel — about half of a ~200 ms page). The next change is therefore a
per-tenant connection pooler, and persistent connections (CONN_MAX_AGE) are withdrawn: they
cannot work under ASGI and leak connections. Full measurement: §8 of
the apparatus.
| Option | Verdict |
|---|---|
| Run the work on the box, report outbound (agent) | Kept — the only model that needs no inbound path, and the same shape as the backup job already running there |
unattended-upgrades on every box |
Kept — security patching with zero central involvement |
| Client-authorised disruptive steps (Matthew's "Microsoft Updates" instinct) | Kept for reboots and major upgrades — the record then shows the schedule, the run and their authorisation |
| Central SSH orchestration over the overlay | Not the default — needs inbound access plus credentials held centrally for every box. Revisit only if a contract demands central control |
| A cloud RMM / vendor patch service | Rejected — puts a third party in the loop and in the data path, and undercuts the sovereign pitch that is the product |
| A signed bundle the client runs themselves | Fallback only — makes compliance depend on a human remembering, and records a gap when they don't |
| Per-client port forwards | Retire — that is the transport project |
| A bigger box to solve an architecture problem | Rejected as a rule |
Measured 2026-10-13 (full table on the capacity plan): primary disk 55% (126/232 GB); app engine is a
3.8 GB box already using 1.99 GB; platform database 1.8 GB of which ~1.65 GB is SNOMED/dm+d reference
data; client metrics ~55k rows for 2 boxes; 29 loopback listeners on the primary; the app opens a fresh
database connection per request (CONN_MAX_AGE = 0).
Unknown — and the reason this page exists:
The first of those is answerable with a scale test rather than an opinion: N simulated client databases and
tunnel peers against the app tier, measuring where it stops being linear. The estate already has a synthetic rig.
Not on this page: DSPT assertion numbers (checked against the current list before anything goes into an
evidence pack), and no secrets or internal addresses.