Status: REVIEW + DESIGN NOTE, 2026-10-14. Reviews the external design session captured at
Scaling SiaaS (a chat with Gemini, 2026-10-13/14 — the full conversation, read
end to end) and turns it into a rig this estate can actually build. Discussion home:
Scaling the Platform; trigger and purchase mechanics:
Platform Capacity & Growth Plan.
Note on the source: the capture restates our own material back at us — the 2026-10-13 baseline numbers, the
decision log, the "buy roles, not capacity" rule and the patching spec's §9 reasoning are ours; the new content
is the test rig, the seeding workflow and the reinforcement table. Read it as one engineer's proposal drawn
from our documents, not as an independent audit.
The methodology is right; the implementation as written would not measure the app tier at all, and could not
run on our hardware. Keep the shape — staged escalation against a knee in the curve, simulated tenants,
pre-agreed thresholds — and rebuild the rig around the two things the session never addressed: how a simulated
client becomes a tenant the application will actually query, and how much simulated load one host can honestly
carry.
| Kept | Why it is right |
|---|---|
| Three-zone model — simulator, ground-truth apps, telemetry | It mirrors what we already have (a proxy rig, live demo tenants, host health telemetry) instead of inventing a lab |
| Phased escalation with a hold at each step | The output we want is the shape of the curve and the point it bends, not one peak number |
| Seed the simulated tenants with realistic data volumes | The best point in the session and one our own plan did not make: empty databases fit in cache and hide exactly the index and I/O behaviour we are trying to provoke. Its follow-on pattern — build one seeded template, dump it, clone it into every simulated tenant — is the right way to do it cheaply |
| Three named scenarios, each with a break metric | Connection/routing (break = connection timeouts and 500s), transport overhead (break = hub CPU saturation, dropped packets), ingestion endurance (break = the central instance slowing global features). Naming the failure signature before the run is what makes the reading decisive |
| The rig must reproduce the transport's real restrictions, not a clean local network | The capture has the simulated clients behind an isolated network mimicking the provider's packet restrictions. Right instinct: a rig on a clean LAN would flatter the architecture |
| Run it combined with the live demo tenants | Matches the request: readings from the existing demo boxes while synthetic load runs behind them |
| Pre-agreed headroom threshold (it proposed 70%) | Maps straight onto the capacity plan's triggers; the number is a ruling, the discipline is correct |
| Build the rig from kit we already own; contingency becomes solution | Same position as §2 of the scaling page — a contingency with a trigger is as good as a solution |
A coupling it surfaced that we should take seriously: the metrics/collection writes land on the same
central instance that carries identity and the knowledge base. Its ingestion test is designed to show a metrics
flood degrading global state and logins. That is a real architectural exposure, and it argues for the collection
host (purchase item 1) even independently of scale.
(a) No tenant registry — the application would never see the simulated clients. A client database is
reached only because a row in the tenant register names it: subdomain, database name, host, port and
credentials, from which the router builds a per-tenant database alias at first use and caches it for the life
of the worker process. Hundreds of tunnel peers with no register rows load the hub and nothing else — the
connection-churn ceiling we are actually hunting is never exercised. This is the single biggest gap: the
session's own best question is when does the app tier stop behaving linearly, and its rig would not touch the
app tier.
(b) Only half the path is drawn. Production has two legs per client: the client box dials out to the hub
(its own database port published as a reverse tunnel), and the app tier separately reaches the hub through its
own tunnel — one forward per client, with a further unit per extra service a client exposes. A rig that models
only the first leg measures the hub, not the application, and its own plumbing hits the same
one-forward-per-client wall as production. Both legs must be in the test, or the test answers the wrong
question.
(c) The sizing is not feasible on this estate. Hundreds of database containers plus hundreds of tunnel
processes is not a workload for an auxiliary host, and not a workload for the app tier. For scale: the
auxiliary host currently runs five containers at 77% disk; the app tier is a 3.8 GB box already at roughly
two-thirds memory with two live clients; the primary carries 26 containers with swap in use. Our existing
"the estate already has a synthetic rig" is a two-container proxy rig (one database, one mock application)
on its own network — an excellent pattern to copy, and nothing like a fleet simulator. Any plan that assumes it
can spawn hundreds of real client instances on that kit is measuring our imagination, not our platform.
(d) The schema, database names and identifiers are invented. The tables, columns, database names, port
ranges and "realistic encryption" of NHS-number-style fields in the session do not correspond to anything in
this estate. Seeding must be driven by the real models (or a clone of a real client database, refreshed and
anonymised), and identifiers stay obviously synthetic — we never fabricate data that looks like a real patient
identifier, however encrypted. The seeded profile is also disproportionate for a shape test: five thousand
patients per simulated clinic is a large fabrication exercise to answer a question about connection and routing
cost, and can be met with a much smaller seeded set once the register and both legs are real.
(e) It breaks two operating rules we hold deliberately. First, dropping monitoring intervals to one second
across the estate: the primary's 2026-09-11 incident was a collector-interval problem, and collection is now
rate-limited on purpose. Second, using the production hosts as the load generator — including the auxiliary
host that serves the test-client library — with "stop the background cron jobs" as a test step. The hub sits in
the data path of every client's database; a synthetic flood through it is a candidate incident, not a test.
Three layers, each of which can be built from kit we own.
1. Load generator — simulated clients, one process per client, not one database per client. Two tiers are
enough to get the truth: a small number of real seeded databases (single figures) for honest index and I/O
behaviour, and N logical clients re-using them, each with its own register row, its own credentials and its
own tunnel leg, for honest connection and tunnel behaviour. Churn and listener counts scale with N; disk
behaviour comes from the seeded pair. That separates the two costs we currently conflate. Seed once, clone —
the session's pattern.
2. Registry and path — both legs, or the test is meaningless. Every simulated client is registered as a
tenant (own subdomain, own credentials, pointing at the rig's database) and given a tunnel leg to the rig hub;
the rig's app-tier instance gets the matching second leg. Worker processes cache the tenant alias map at first
use, so the rig's run script must start the app tier after the register is populated — a detail that will
otherwise silently produce a flat, meaningless curve.
3. Telemetry — persist first, capture second. Running the rig before platform metrics are persisted
produces window-only readings that cannot be trended or repeated, and therefore cannot set a trigger. Persist
platform host metrics into the app (the scaling page's first action), then add per-run captures: request
latency percentiles, database connections and connect time, hub CPU per tenant, central-database write rate and
lock waits, and background queue depth. The session's suggestion of pairing each run with a control-group
measurement on the live demo tenants — watch their query latency drift while the synthetic fleet churns — is
the right instrument and should be kept.
Isolation rules — non-negotiable. The rig runs on its own network and port range; it gets its own
app-tier instance, not production Daphne; it does not run through the production hub while real clients are
on it; runs are scheduled; one kill switch stops generator and tunnels; monitoring intervals are left alone.
| Stage | Simulated clients | Real seeded databases | What it is for |
|---|---|---|---|
| 1 · Baseline | the two live demo tenants | 2 | Ground truth: latency percentiles, hub cost per tenant, connection profile per clinic |
| 2 · Ramp | 5 → 10 → 25 | 3 | Per-tenant cost and where the curve bends — this stage contains the transport trigger |
| 3 · Stress | 50 → 100 | 5 | Connection churn, hub CPU, central-database writes; extrapolate to 300 rather than build 300 |
| 4 · Peak event | 100, plus simultaneous telemetry ingest and imaging traffic | 5 | Deliberately find which component fails first, not whether one does |
Building 300 real client instances is exactly what the model says we cannot afford — and that is itself a
finding. Measure the curve, state the extrapolation, and label it as an extrapolation. Run each stage against
the three named failure signatures from §2, so a stage is only "passed" when none of them fires.
The overlay (lever 1 of the scaling page) is the only lever whose fix is a project rather than a purchase, so
the gate was to prove that inbound UDP reaches the hub before four to eight weeks were committed. It does
not reach the hub.
Method. The primary's firewall already permits UDP on the overlay ports from anywhere. Datagrams were then
sent from two unrelated networks (the app engine and the auxiliary host) to the primary's public address on a
permitted port and on an unpermitted one, with a capture running on the primary; both sending hosts were
controlled for, by resolving DNS over UDP in single-digit to tens of milliseconds and reaching the primary over
TCP in 10–25 ms.
Result: zero datagrams arrived on either port from either network, while TCP from both hosts answered in
milliseconds. Corroboration from inside the estate: the WireGuard interfaces already standing on the primary
and the app engine — peers configured, keepalives every 25 seconds, enabled at boot — report 635 KiB sent and
0 bytes received, with no handshake ever completed.
Conclusion: inbound UDP to the primary is filtered upstream of the host (the host's own firewall permits it
and the packets never appear on the wire). Three consequences:
Also found: a half-built overlay that has never passed a byte — two interfaces, keys including a
per-client key, firewall rules, a tuned MTU and forward rules, deployed on two hosts and silent. It raises no
alert, so nothing surfaced it. That is worth a monitoring item whatever transport wins, and it is a reminder
that a deployed-but-dead path looks identical to a healthy one from the dashboard.
| Decision | State |
|---|---|
| App-tier fixes (connection reuse, worker processes, staggered polling) | Agreed. The staggered poll is deployed and verified (§4 of the apparatus page); the two app changes are raised as a work order to the engineering agent |
| The reachability gate | Run — failed. See §6 |
| The collection host (purchase item 1) | Decided: do not buy yet. The measured case for it was a connection cliff (now closed by the staggered sweep) and the coupling between metrics writes and identity/the knowledge base. Both are addressable without a purchase; fund it when persisted metrics show collection's share of the primary climbing, at roughly fifty clients, or when the rig needs a telemetry home — whichever comes first. This supersedes "the cheapest big reliability win" in the capacity plan, and that page's purchase order should be updated to match |
| Stage sizes and where the rig lives | Open — see below |
The rig question, in plain terms. Building the rig raises two practical choices, and neither needs money:
The only case for touching a production host is wanting absolute throughput figures for the production container;
that is a second pass in a window, not the first one. Stage sizes stay as in §5 (5 → 10 → 25 → 50 → 100), and
they can be run in any order once the first stage proves the plumbing.
Housekeeping note: infrastructure/scale_siaas holds the full conversation but is stored under a
placeholder title with no description and is not linked from the scaling page; it should be retitled,
described and cross-linked (or archived under the scaling tree) once this note is agreed.
Not on this page: DSPT assertion numbers, and no secrets or internal addresses.