Status: pooler shipped 2026-10-14 (§9) — the connection-setup plateau moved and now scales with concurrency. §4 and §5 were revised when the app-tier work order falsified the serialised-process premise; where §8/§9 (the engineering agent's measured write-up) and the earlier text disagree, §8/§9 win. APPARATUS LIVE + MODEL RUNNING, 2026-10-14. First working answer to "when and where does
the infrastructure buckle as clients are added". The apparatus measures the per-client coefficients; the
model turns them into break-points. Design rationale and the review of the external session:
Scalability Testing Ground — Design Review. Discussion home:
Scaling the Platform; trigger and purchase mechanics:
Platform Capacity & Growth Plan.
We do not need to build three hundred client boxes to know where the estate breaks. Almost every cost in this
platform is linear in the number of clients with a measurable coefficient, so the honest method is:
That is the apparatus: measurement, model, validation — in that order.
Two instruments, both in the ops repository under scripts/, both runnable on the primary read-only:
| Instrument | Modes | What it measures |
|---|---|---|
scale-measure.py |
connect |
TCP connect latency to each published client port. Proves the hub accepts client ports locally, so a connect measurement hides the transport — a false reading removed before it misleads anyone |
roundtrip [--compare] |
A full request/response over a client tunnel, and with --compare the split between a fresh connection (p50 66.6 ms) and an established one (p50 7.0 ms) |
|
cpu |
Hub CPU per new vs reused connection, whole-host and per tunnel session — shows the tax is latency, not CPU (~0.75 ms of sshd CPU per connection) | |
scale-model.py |
[N], --constants |
The maths: measured constants + N → break-points per component. Runs in seconds, so the model is re-derived rather than re-argued when a constant changes |
Run scale-measure.py <mode> with a small sample count first; the full run is the defaults. Both are
read-only — connects, unauthenticated GETs answered with 403, /proc and /proc/stat reads.
A third script is the first fix rather than an instrument: collect-capacity-sweep.py replaces the
one-cron-line-per-client poll with a single staggered sweep (§4), and is deployed on the primary.
| Constant | Value | How measured |
|---|---|---|
| One round trip over a client tunnel | 7.0 ms p50 (min 6.8, p95 7.3) | 120 requests on one established connection |
| Cost of a fresh connection | 66.6 ms p50 — i.e. 59.6 ms of pure setup tax | 120 requests, new connection each time |
| Hub sshd CPU per new connection | 0.75 ms (0.25 ms reused) | Per-session /proc jiffies across 400-connection phases |
| Private memory per client tunnel session | 15 MB (Pss 14.8–15.5 MB) | smaps_rollup on the sessions carrying client ports |
| Capacity rows per client per day | 96 (15-minute poll) | 192 rows/day currently recorded for two boxes |
| Tunnel-health rows per capacity row | 2.09 | Measured ratio in the central tables |
| Bytes per row (incl. indexes) | 203 B capacity, 106 B tunnel | Table size ÷ live rows |
Central instance max_connections |
100 (9 in use) | SHOW max_connections, pg_stat_activity |
| Application requests served today | 147 app requests in 11 h ≈ 0.004 req/s estate-wide — ~1 request per 8 minutes per demo tenant | nginx access log, app paths only, 2026-10-14 |
| Per-page database work | 1 central + 2 tenant queries, and 2 fresh connections per page | measured in the running app, 2026-10-14 (§8) |
| Tenant connection, fresh vs reused | 55.7 ms p50 fresh, 7.0 ms p50 reused → 48.6 ms tax | the engineering agent's db instrument, 2026-10-14 |
| Measured throughput | 25–30 req/s for a tenant page, 65–69 req/s for a trivial DB endpoint, flat at c10 and c40 | ab -k from the primary, 2026-10-14 (§8) |
| Web container | 2 vCPU / 1536 MB, gunicorn + 2 uvicorn workers since 2026-10-14 | Compose limits and the container's own process list |
| Per-request database connections | 1 fresh connection per request | CONN_MAX_AGE = 0 in the tenant alias map, and Django's default for the central alias |
| Async views in the application | none — all sync, but they do not serialise: each request gets its own thread | 403 sync views; Django 5.2's ASGI handler enters a fresh ThreadSensitiveContext per request and asgiref gives each context its own executor (§8) |
Hub memory is not the first problem. 15 MB per client session puts the primary's memory at ~52% at 300
clients, crossing the 70% trigger only at N ≈ 498. The published ports (≈450), the cron lines and the
credential files (one set per client) are plumbing work, not a cliff.
The collection plane broke at ~90 clients — fixed 2026-10-14. The poll was one cron line per client on
the primary, each opening its own connection to the central instance, all firing in the same minute four times
an hour: at N ≈ 91 the central instance's 100-connection limit would have been exhausted and polls refused.
It is now a single staggered sweep — one cron line for any number of clients, each client assigned a slot
derived from its own slug, the client list held in a manifest — deployed and verified on 2026-10-14 (a real
snapshot row landed through the new path). The cliff is closed before the client count could reach it, and no
cron edit is needed when a client is added.
Metric volume is a time bomb, not a wall. ~297 rows and ~40 kB per client per day. At 300 clients that is
~89,000 rows/day — the capacity table reaches the 5M-row trigger in 56 days. Retention and rollup therefore
have a deadline attached to the client count, which is a far more useful statement than "do it soon".
The application tier is where it breaks, but not for the reason first modelled. The first version of this
page argued that one daphne process serialised every request, because all views are synchronous. Measurement
falsified that (§8): Django's ASGI handler runs each request's sync view in a fresh ThreadSensitiveContext,
and asgiref gives each context its own single-thread executor — so requests run in parallel, one thread
each, and the worker count is not the constraint. What is:
What this does to the client headcount. Take the measured ceiling and the demand range from §3:
| per-client demand | clients at 25–30 req/s (tenant pages) | clients at 65–69 req/s (trivial writes) |
|---|---|---|
| 0.002 req/s (measured demo floor) | ~12,000–15,000 | ~32,000+ |
| 0.1 req/s (30 staff, one request each per 5 min) | 250–300 | 650–690 |
| 1.5 req/s (the model's placeholder) | 17–20 | 43–46 |
So the honest summary changed: the estate is not limited to ten clients by a serialised process; it is
limited by whatever serialises in the connection-setup path, and at a plausible practice demand it is somewhere
around a few hundred clients — with the traffic shape (how many pages per user per hour) doing all the work
in that estimate. The gain available from fixing the setup path is roughly a 7× reduction in per-request latency
(55.7 ms → 7.0 ms per connection), which moves every row of that table up.
| # | Fix | Effect | Cost |
|---|---|---|---|
| 1 | Per-tenant connection pooler (item 6 before this revision) — central + each tenant | removes both problems at once: the 48.6 ms fresh-connection tax and the ASGI non-reuse, which is what caps throughput today | Project — needs a per-database mapping and an auth design (engineering agent's assessment) |
| 2 | CONN_MAX_AGE) |
cannot work under ASGI (per-request thread ⇒ per-request connection) and leaks connections — with 60 s set, ~150 requests exhausted the central instance's connection limit. Left at 0, env hook only | Withdrawn |
| 3 | Multiple app worker processes — done 2026-10-14, container raised to 2 vCPU / 1536 MB | does not move the ceiling (measured flat across 1 and 2 workers); kept for packaging and headroom | Done |
| 3 | Stagger or batch the collection poll | done 2026-10-14 — the ~90-client connection cliff is closed; one cron line regardless of client count | Hours |
| 4 | Persist platform metrics (Action A) | every trigger becomes a trend instead of a snapshot | Days |
| 5 | Retention + rollup on the metrics tables | keeps the 5M-row deadline from arriving | Days, and cheaper now than later |
| 7 | Transport overlay | plumbing, per-request latency, session memory — and it gets harder with every client. Its mechanism is blocked: see the gate result | Project, 4–8 weeks — re-scope first |
| 8 | Second app engine + load balancer | scale-out once one box is full | Purchase, 2–4 weeks |
The ranking changed on 2026-10-14, and the reason is worth keeping: two of the three configuration answers turned out to be wrong or ineffective once measured, and the item that was ranked sixth is now first. Measurement, not the argument, decided the order. Only the pooler is a project; everything after it in the list is either a purchase or a transport decision.
The model is only as good as its assumptions, and four of them are unknowns, not measurements:
The rig is therefore for validation and for the unknowns, not for discovering the whole curve: register
simulated tenants, build both tunnel legs, run them against a rig app-tier instance, and measure those four
numbers. Stage sizes (5 → 10 → 25 → 50 → 100 with a few real seeded databases) and isolation rules are in the
review page; the build is app-side work (tenant registration, worker processes, pooling) plus ops-side work
(generator, tunnels, telemetry).
The coefficients were measured against the demo client boxes over the public internet, so the round-trip figure
carries their uplink quality; the setup tax (59.6 ms) looks like a TCP small-packet/delayed-ack interaction on
the tunnel and should be re-measured with connection reuse in place. Row sizes are current values and will drift.
The staff/request assumptions are placeholders. And every break-point above is a prediction from measured
constants — the rig exists to falsify it, which is exactly the point of building it.
Not on this page: DSPT assertion numbers, and no secrets or internal addresses.
Django's ASGI behaviour was tested directly after §4 was written, not assumed. The result falsifies the
"single serialised thread" premise in §4 and moves the binding constraint from the web process to the
per-request database connection setup.
Instruments. scale-app-measure.py (new, ops repo): db times a tenant query with a fresh vs a reused
connection inside the running app; session mints a real DB-backed login cookie; http fires N requests
from C concurrent clients. End-to-end throughput was cross-checked with ab -k from the primary.
Finding 1 — CONN_MAX_AGE cannot reuse connections under ASGI, and leaks them. Django's ASGIHandler
runs each request's sync view inside a fresh ThreadSensitiveContext, i.e. a new thread per request
(__aexit__ shuts the single-thread executor down at request end). Django stores DB connections per thread,
so every request opens a fresh connection anyway — verified directly: four successive request-shaped
contexts landed on the same recycled thread id but started from connection is None each time, while calls
without a per-request context reused one. With CONN_MAX_AGE=60 the orphaned per-request connections then
accumulate: under ~150 requests the central instance returned FATAL: sorry, too many clients already
(max_connections=100). CONN_MAX_AGE is therefore left at 0 for the web tier.
Finding 2 — the connection tax dominates, and it is two connections, not one. A tenant page issued only
1 central + 2 tenant queries, yet took ~200 ms: each request opens a fresh connection to the central
instance (tenant/session lookup) and a fresh connection to the tenant database, and each setup costs
~50 ms over the tunnel (db mode: fresh p50 55.7 ms, reused p50 7.0 ms → 48.6 ms tax). Roughly
half of a request's ~200 ms is connection setup, not query work.
Finding 3 — worker processes do not move the ceiling. Throughput is flat across server and process
count (ab -k, tenant page, c10): daphne 1 process 25.3 rps, gunicorn 1 worker 26.6 rps, gunicorn 2
workers ~24.8 rps; at c40 all sit at 27–30 rps. A trivial DB-touching endpoint
(/infrastructure/health/) reaches only ~65–69 rps because it too opens a fresh central connection per
request. The web process is not the current bottleneck; the connection-setup path is.
Shipped 2026-10-14 (gp_booking_app ddd92a8, 4472c8e): gunicorn + uvicorn.workers.UvicornWorker,
WEB_CONCURRENCY=2, container raised to 2 vCPU / 1536M (packaging and headroom); DB_CONN_MAX_AGE env hook
defaulting to 0. Verified live: health 200, protected pages 302 → OIDC, WebSocket echo works, central
max_connections steady at 9. Not shipped: persistent connections (Finding 1).
Corrected model. Replace "one process serialises every tenant" with "every request pays ~2 × 50 ms of
connection setup, and the app worker count is not the constraint until that is fixed". The next
reinforcement is now unambiguously §5 item 6 — a per-tenant connection pooler — the fix for both the tax
(Finding 2) and the ASGI non-reuse (Finding 1) — ahead of adding more app workers.
The pooler from §5 item 6 is live on the GP app tier: pgbouncer (transaction mode) on the app host holds the
tunnel-side connections open, and the web process opens a cheap local connection to it. It fixes both §8
Finding 1 (ASGI does not reuse connections) and Finding 2 (the ~2 × 50 ms per-request setup).
Shape. One pgbouncer service on the app host; pool_mode=transaction, default_pool_size=6,
max_client_conn=400. It routes by database name, and every [databases] entry carries that tenant's own
credentials. The web process authenticates to it as a single generated user (veripath_pooler, md5) and
selects a database by name; tenant DB credentials are not in the pooler's userlist and authenticate
nothing. Celery and migrations stay on the direct path (the web service sets DB_VIA_POOLER=1; migrations run
with DB_VIA_POOLER=0, since transaction pooling is incompatible with Django's migrate advisory lock). The
config is generated from the runtime settings and the tenant register by manage.py generate_pooler_config
(0600, host-local, reloaded via pgbouncer's admin console) — never hand-maintained.
Before / after — same instrument, same session, tenant page /appointments/, ab -k from the primary:
| before (direct) | after (pooler) | |
|---|---|---|
connection setup per request (db mode, p50) |
48–78 ms | 1.09 ms |
| throughput, c10 | 29.1 req/s | 50.8 req/s |
| throughput, c40 | 31.1 req/s | 74.1 req/s |
| sustained c50, 2000 requests | — | 74.2 req/s, 0 failures; central connections steady at 16 |
The plateau did not merely move — it now scales with concurrency (29→31 before, 51→74 after), evidence that
the constraint was the connection setup and not the tunnel. §8 Finding 1 did not recur: central connections
held at 16 through the sustained run, against >100 (refused) under CONN_MAX_AGE=60.
What is not claimed. The transport round trip (~7 ms/query) is unchanged — the pooler removes setup, not
per-query latency, so a query-heavy page still pays the tunnel per query. The pooler is GP-only for now;
dental and pharmacy are unchanged. Pool size and worker count were left at the conservative defaults above;
default_pool_size is the next lever if a single tenant's page mix grows.
Re-measured from scratch with the same instrument and the app's own code paths, rather than re-read. All figures are
mine, taken after the change; the direct-path pair was taken in a single docker exec with DB_VIA_POOLER=0, so
it never touched the running app.
| check | result |
|---|---|
tenant-query setup, direct path (DB_VIA_POOLER=0, that process only) |
fresh p50 55.99 ms, reused p50 6.99 ms → tax 49.0 ms |
| the same, through the live pooler | fresh p50 8.62 ms, reused p50 7.15 ms → tax 1.47 ms |
ab -k, tenant page /appointments/, c10 (200 requests) |
51.7 req/s, 0 failed |
ab -k, same page, c40 (800 requests) |
78.3 req/s, 0 failed |
| central connections sampled every second (110 s, 1,000 requests inside it) | idle 6, 11 through the load window, then 12 → 16 when the alias sweep below opened a pool per database (each lingers for server_idle_timeout = 120 s) — bounded throughout against max_connections 100, no growth on repeated requests: §8 Finding 1 did not recur |
| every alias the web app can reach | 7/7 via the pooler, 0 failures — default, sector_gp, sector_dental, sector_client, and the three client-sector tenants (test-client, test-client-dental, test-client-pharmacy) |
| wrong password / a tenant DB username / an unlisted database | all three refused (password authentication failed, password authentication failed, no such database) |
| the listed database with the pooler user | reachable (SELECT 1) |
| config posture | directory 0700, files 0600, owner 70:70, mounted read-only into a container running as uid 70; userlist.txt carries only the pooler user; every [databases] entry carries a forced user and password |
Why the alias check matters. The generator emits one entry per database the web app can reach (static aliases plus
client-sector tenants, is_active only) — the same filter the dynamic-alias loader uses — so the [databases] list is
co-extensive with the alias set: there is no tenant the app can route to a database pgbouncer does not know, and no
entry the app cannot reach.
Residual observations — none of them a defect, all worth knowing. (1) The pooler user is a single key to all five
databases: by name it reaches every listed database (verified above), so the isolation boundary is now the app's
alias routing plus that one credential, where the direct path had a credential per tenant. The per-tenant credentials
themselves are no longer presented by the app at all. (2) admin_users and stats_users are set to the same
credential the app authenticates with, so the web process can RELOAD or PAUSE the pooler. (3)
log_connections/log_disconnections are 0, so the pooler keeps no per-connection record; setting them to 1 costs
one line per connection and puts it in the container log, which is the kind of evidence an auditor asks for. (4) The
pooler credential lives in the web container's environment, as it must to authenticate — see the credential-handling
note in housekeeping Round 53.