Status: PLAN, 2026-10-13. Written because the client-VPS patching scope
(Client VPS Patching — Managed Service) commits us to touching
every client box on a schedule — and because we cannot demonstrate that the platform tier holds at 200–300
clients. The position this plan takes is Matthew's: assume it will not hold until measured, and pre-plan the
steps and the signs that tell us when to act — a contingency with a trigger is as good as a solution.
The wider scaling question has its own discussion home — Scaling the Platform
(the levers, the decision log, what is measured vs unknown); this page is its trigger-and-purchase half.
| Component | Today | Note |
|---|---|---|
| VPS1 (primary/orchestrator) | disk 126 / 232 GB (55%), RAM 4.7 of 16 GB, load 0.87 | 29 loopback listeners and 3 tunnel/tenant-gateway units today — the count that grows with every client box |
| VPS2 (app engine) | disk 28 / 116 GB (24%), RAM 1,991 of 3,852 MB, load 0.16 | the app tier is a 3.8 GB box; app containers ~1 GB total, each with a 1 GiB limit |
| VPS3 (auxiliary) | ERPNext + redis, inside the monthly cadence | no patch/capacity work pending |
| Platform database | 1.8 GB, of which ~1.65 GB is SNOMED / dm+d reference data | the reference plate is static; growth comes from operations and metrics |
| Client metrics | capacity snapshots 17,983 rows, tunnel health 37,410 rows — for 2 client boxes, ≈192 snapshots/day (≈94 days of history) | the row-volume driver, not the byte-volume one (yet) |
Projection: 300 clients at 15-minute polling is ~28,800 snapshots/day ≈ 10.5 million rows/year. At a
few hundred bytes a row that is single-digit GB per year — survivable, but only if retention and rollup are
decided before the table is large rather than after.
PlatformVPSHealth has 0 rows. The three platform hosts' health is published as JSON snapshots
(platform-vps-health.sh → synced to the app engine), so we can see now but have no queryable trend for
the platform tier. We also record no per-client cost.
Until that changes, every trigger in §5 is a fire drill rather than a plan: we would learn we are near a limit
by feeling it.
Action A (cheap, start now): persist the platform hosts' metrics into the app — the model already exists —
and record the per-client cost ratio (central-DB bytes, app CPU/RAM/connections, tunnel bandwidth, logins).
That single change converts "no crystal ball" into a measured slope, and it also feeds the per-client price.
We do not have to guess how this behaves at N clients — the estate already has a synthetic rig and a cold
standby. The honest test is N simulated client databases and N tunnel peers against the app tier, then
measure the shape rather than the number:
CONN_MAX_AGE = 0 connection churn (gp_booking_app/tenancy/routers.py:84) — with aCommit: run the scale test before the first ten real clients, so the first purchase decision is informed by
a curve rather than an opinion. The output is a short measured report with the ceiling stated.
| Signal (where measured) | Act when | Action | Lead time |
|---|---|---|---|
| Central DB disk / CPU, autovacuum pressure | disk >60% or sustained CPU with bloat | split the metrics store onto its own Postgres instance | 2–4 weeks |
| Metric row growth | tables >5M rows or >500k rows/month | retention + rollup (raw 30–90 days, hourly/daily beyond) | ~1 week to build |
| App tier RAM / latency (VPS2, 3.8 GB) | RAM sustained >60%, any container at its 1 GiB limit, or p95 latency rising | second app engine + load balancer (tenant routing already keys off the hostname) | 2–4 weeks |
| Connection churn (client DBs) | connections/sec climbing, or connect time visible in app timings | set a sane CONN_MAX_AGE with pooling |
days |
| Transport shape (primary) | >10 client boxes (forwards/listeners/units), or hub bandwidth >40% of link | WireGuard overlay migration; retires the per-client forwards | 4–8 weeks — a project |
| Collection load on the primary | monitoring share of CPU climbing (the 2026-09-11 incident was this) | move collectors off the primary | 1–2 weeks |
| Backup staging (VPS1 disk 55%) | overlapping client backup windows, staging pressure | dedicated staging host | 2–4 weeks |
| Identity (Keycloak) | login latency, or realm sprawl unmanageable | dedicated identity host | 2–4 weeks |
The transport trigger is the one to watch hardest, because it is the only one whose fix is a project rather
than a purchase — and it gets harder with every client added in the meantime.
Each buys a role rather than raw capacity, and they are ordered by value per pound:
capacity-monitoring, platform-vps-health, tunnel-health,backup-health) feed the dashboard; the client boxes feed the exceptions list in the patching spec.Triggers are only as good as their trend data (§3). The scale test is a model, not the real estate — clinic
hours, month-end and vaccination season load differently. And two items cost more to delay than to do: the
metrics retention/rollup and the overlay, because both get harder with every client added.
Not on this page: DSPT assertion numbers, and no secrets or internal addresses beyond what the estate's own
documentation already carries.