Living record of capacity reviews of the VPS1 root filesystem. Each pass appends a dated
section: findings, what was reclaimed, what was made permanent, and what is still open.
Read together with Backup Overview and the
Operations Guide.Method: measure every candidate before touching it; reclaim only what is regenerable
or superseded; never change a backup, monitoring or security posture without an explicit
decision recorded here. Restore-path changes are called out, because they change how a
restore is performed.
Trigger. Matthew reported that free space "is beginning to diminish quite a bit each
day", with capacity otherwise fine.
Starting position. 149G used of 232G (65%), 83G free.
Three separate things had been conflated, and only one of them was a leak:
1. A kernel audit flood — the primary daily drain (≈150 MB/day).
A single AppArmor record was being written to syslog, kern.log and the journal
continuously and indefinitely:
apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default"
pid=<netdata apps.plugin> comm="apps.plugin" requested_mask="read" denied_mask="read"
The apps.plugin process inside the Netdata container (whose job is per-process metric
collection) repeatedly attempts a ptrace that the container's docker-default AppArmor
profile denies. It never succeeds, and it never stops trying.
Measured: syslog was 991,130 denial lines / ~1.05 GB in one rotation period; sampled
baseline immediately before the fix was 158,400 denial lines/day and 172 MB/day into syslog
alone, with the same records again in kern.log (≈45 MB/day) and the journal.
Consequences beyond raw bytes: it inflated the journal (4.0G), and the journal had no
size cap configured — the systemd default is 10% of the filesystem, i.e. it was free to
grow towards ~23G.
2. Uncompressed database dumps — large, but a steady state, not a leak.
/var/backups/daily holds 36G. The two largest streams are written uncompressed:
| Dump | Per day | Held (7-day rotation) |
|---|---|---|
defectdojo_pg_*.sql |
3.55 GB | 27 GB |
gp_booking_pg_*.sql |
0.85 GB | 5.9 GB |
wikijs_pg_*.sql.gz |
0.035 GB | (already compressed) |
forgejo_pg_*.sql |
0.9 MB | negligible |
volumes/*.tar.gz |
0.37 GB | 3.0 GB |
≈4.4 GB/day is written, and primary-backup.sh retains 7 days (-mtime +7 -delete), so
the footprint plateaus rather than growing without bound. It is the single largest daily
disk consumer on the host, and roughly ten times what it needs to be — the same script
already gzips the wikijs dump, so the pattern is established; the two biggest streams were
simply never switched over.
3. A crash-looping legacy dev service — a second, independent drain (≈126 MB/day).
Surfaced by the verification sampling for (1), not by the original sweep. See "New finding"
below.
Everything below is regenerable or superseded. No backup, monitoring or security setting was
weakened, and no service was disrupted (verified after: every container healthy, wiki and
public site both HTTP 200).
| Action | Reclaimed |
|---|---|
journalctl --vacuum-size=1G |
3.0 GB |
Forced logrotate on the flood-swollen syslog/kern.log (2 cycles, so delaycompress completed) |
~2.6 GB |
Truncated /var/log/dmesg — 395 MB, stale since 2026-09-22, matched by no logrotate rule, rewritten at boot |
395 MB |
/root/.cache/trivy (2.8 GB) + /tmp/trivy-scans (738 MB) — scan DBs and scratch, re-fetched by the weekly scan |
3.5 GB |
docker builder prune -af (build cache, unused) |
1.47 GB |
npm _cacache for root (525 MB) and matthew (768 MB) |
1.29 GB |
Stale /tmp scratch (files >14 days, mockup/demo dirs) + apt-get clean |
~1.5 GB |
⚠️ One retention side-effect, recorded deliberately: the journalctl --vacuum-size=1G
trimmed journal history older than the last ~1 GB. The lost history was overwhelmingly the
audit flood above. If an investigation later needs older journal entries, they are gone — the
rotated syslog.*.gz files (kept, 4 generations) are now the deeper history.
1. Journal capped at 1G — drop-in /etc/systemd/journald.conf.d/99-veripath-cap.conf
(SystemMaxUse=1G). Previously uncapped, with a ~23G ceiling derived from disk size.
2. A size ceiling on every rsyslog file — /etc/logrotate.d/rsyslog gains
maxsize 100M (previous file kept as rsyslog.bak-20260926). Retention unchanged at 4
generations; the change only bounds how large one file may get before it rotates. No single
log file can now exceed ~100M, whatever a source does — the structural reason this cannot
repeat even if a new flood appears.
3. The audit flood stopped at source — DECIDED AND IMPLEMENTED 2026-09-26.
Matthew approved the recommended option (a): [plugins] apps = no in
/opt/infra/netdata/netdata.conf, with the netdata container restarted to pick it up.
Verified in a 120-second window after the restart:
| Measure | Before | After |
|---|---|---|
New type=1400 audit lines in syslog |
~110/min (158,400/day) | 0 |
New apps.plugin lines in kern.log |
continuous | 0 |
apps.plugin processes in the container |
1 (looping) | 0 |
Nothing of value was lost: the ptrace was denied, so those charts were never being
populated from it. Only the per-process / per-user "Applications" charts go; system, CPU,
memory, disk, network and the Docker container metrics (via go.d) are unaffected. Netdata
reports healthy and its API answers 200 after the restart.
The alternative — security_opt: [apparmor=unconfined] on the netdata service — was
rejected: it would allow the ptrace and keep the charts, but it weakens container isolation
on a container that already mounts the Docker socket and runs on the host network.
Config is version-controlled and pushed: /root/work/veripath-ops @ 4f50d20
("ops: disable Netdata's apps.plugin on VPS1 (kernel audit flood)"). Revert by setting
apps = yes and restarting the container.
4. /var/log/dmesg truncated and noted: it is covered by no rotation rule. Worth folding
into a rotation config at the next ops change.
gp-booking-dev.service crash loopThe verification sampling for (1) showed syslog still receiving ~175 KB/120s after the
audit flood stopped. All of it is one service.
/etc/systemd/system/gp-booking-dev.service — "GP Booking App (Dev Server on portenabled, Restart=on-failure, RestartSec=10, running a/root/gp_booking_app_dev.ModuleNotFoundError: No module named 'channels' — the dev venv has nochannels package (0 matching entries in its site-packages). The unit last servedRestartSec=10 sits exactly at the boundary of systemd'sStartLimitIntervalSec=10s / StartLimitBurst=5, so five restarts in ten seconds isCost: ≈126 MB/day of log, a repeated gunicorn worker spawn every 10s, and ~0.86 s of CPU per
attempt — for a service that raises nothing and answers nothing.
ACTIONED 2026-09-26 — stopped and disabled. Approved by Matthew the same day;
systemctl disable --now gp-booking-dev.service executed and verified.
| Measure | Before | After |
|---|---|---|
New syslog lines / 90 s |
~1,095 | 13 |
New syslog bytes / 90 s |
~134,627 | 4,870 (≈4.7 MB/day) |
gp_booking_app_dev processes |
looping | 0 |
| Port 8080 | nothing listening | nothing listening |
Nothing depended on it — systemctl list-dependencies --reverse returned only the default
multi-user.target → graphical.target chain, nothing referenced it by name, and no nginx or ops
config proxies to port 8080. VPS1 is documented as running no GP booking runtime; the app lives
on the app engine and VPS1 keeps only the development source copy.
Disabled and stopped, NOT deleted — the unit file remains at
/etc/systemd/system/gp-booking-dev.service, so the change is one command to reverse. The unit and
the reasoning are preserved in the ops repo:
/root/work/veripath-ops @ d47e648 (systemd/gp-booking-dev.service.disabled plus
…disabled-note.md, which carries the revert steps and the rate-limit fix to apply if the unit is
ever wanted back).
Lesson worth keeping: the unit looped forever because RestartSec=10 sits exactly at the
boundary of systemd's default rate limiter (StartLimitIntervalSec=10s, StartLimitBurst=5) — five
restarts inside ten seconds is never reached, so the burst limit never engages and the unit never
settles into failed. Any unit with Restart=on-failure, RestartSec >= 10s and no explicit
StartLimit* will do this. Check every unit of that shape.
Both floods are now stopped. Combined log growth from the two sources was ≈295 MB/day
(172 + 123); it is now ≈5 MB/day. That was the drain Matthew noticed.
Approved by Matthew and completed the same day. primary-backup.sh now streams every database
dump through gzip -6 at nice -n 10 — the two large streams because that was the point, and the
three small ones so the restore rule is uniform. Committed and pushed:
/root/work/veripath-ops @ 4eb099b.
Measured on a real run (3 min 12 s, 2026-09-26 08:57):
| Stream | Before | After | Ratio |
|---|---|---|---|
defectdojo_pg |
3.30 GB | 827 MB | 4.3:1 |
gp_booking_pg |
810 MB | 127 MB | 6.7:1 |
wikijs_pg |
33 MB | 35 MB | already gz |
forgejo_pg |
891 KB | 125 KB | 7.1:1 |
host_pg |
282 KB | 30 KB | 9.5:1 |
| Total written/day | ≈ 4.4 GB | ≈ 1.0 GB | |
| 7-day footprint | ≈ 33 GB | ≈ 7 GB |
The estimate in the original finding was optimistic and is corrected here: DefectDojo compresses
at 4.3:1, not the ~10:1 assumed, so the win is ≈1.0 GB/day and ≈7 GB standing rather than
≈0.5 GB/day and ≈4 GB. Still a 4.5× reduction, and by a wide margin the largest capacity win of
this pass.
Robustness gain worth recording. Rather than a one-line pipe, each dump streams into a .tmp,
is verified with gzip -t, and is only then moved into place — a truncated dump can never be
published as a good backup. set -o pipefail was added because previously a dump command that
failed while piping into a successful gzip was logged as OK (the wikijs line had this weakness),
which is the worst possible failure mode for a backup.
Verified, not assumed:
gzip -t clean on all five streams, run independently of the script.gp_booking_pg CREATE DATABASE set is byte-identical to the previous uncompressed dumpPostgreSQL database cluster dump complete marker is present in each pg_dumpall stream.wikijs_pg dump was restored into a scratchpsql exit 0, zero stderr, ON_ERROR_STOP set — and 548 pages restored againstbackup-healthcheck.sh both confirmed still working afterwards.Docs updated in the same change, as required (the restore path changed, so the runbooks move with
it) — Backup Overview and
Quarterly Restore Drill. Both were badly
stale beyond the compression and are now correct; the drill page also had the central database
password written as a literal in six commands and referenced the pre-rename container
3ff9c011e629_gp_booking_postgres. Both removed — see the security finding below.
Found while updating the restore drill. infrastructure/backups/quarterly-restore-drill carried the
central PostgreSQL password as a literal string in six shell commands. It has been removed
from the page (replaced with the host-local 0600 config read that the backup script already uses)
and the redaction was verified against the rendered page.
⚠️ Redaction is not remediation. The value was readable to anyone with wiki access and remains
in the page's revision history, so it must be treated as disclosed and rotated. The
procedure exists: Central Database Password Rotation — Runbook.
This is a second instance of the class already open on the housekeeping log as "secrets sweep
needed" (alongside development/local-agent-instructions), which strengthens the case for sweeping
the whole wiki rather than fixing pages as they are stumbled over.
Each is either an asset, evidence or a judgement call:
| Item | Size | Note |
|---|---|---|
local_hermes_agent stopped container |
28.7 GB | Writable layer of a container stopped 2026-08-09. Largest single item on the host. Not a documented rollback asset; contents not yet reviewed. |
defectdojo_stage_defectdojo_postgres volume |
4.6 GB | Unused (0 links). Likely the documented pre-move DefectDojo rollback volume — do not remove without confirming. |
/root/forensics |
880 MB | Incident evidence (repos/backup/tmp). |
/root/dd-move |
652 MB | DefectDojo move artefacts (2026-09-13). |
/var/backups/migration-images + premigration |
2.1 GB | 2026-08-11 migration bundle (apps-images.tar.gz, defectdojo-images.tar.gz, config + pre-migration dump). |
Retired-on-VPS1 dental images, incl. ollama/ollama |
~11 GB | Dental runtime retired from VPS1 on 2026-09-11; container layers small, but the images stay resident. |
Disk alerting already exists and was not touched: /opt/scripts/platform-vps-health.sh
collects disk_usage_pct every 5 minutes into the status JSON synced to the app engine (VPS2).
So the "keep an eye on capacity" requirement is met by existing monitoring — this pass restores
the headroom, it does not add the watchdog.
All running containers healthy, no restarts/unhealthy. systemd-journald, rsyslog, docker,
nginx all active. Wiki 200, public site 200. Journal 4.0G → 924 MB. /var/log 7.6G → 1.5G.
Root filesystem 149G → 137G used, 96G free (59%).
Both root causes are now stopped and verified by before/after counts over fixed windows:
| Source | Daily log growth | Status |
|---|---|---|
Netdata apps.plugin AppArmor ptrace denials |
~172 MB/day | 0 new denials in 120 s |
gp-booking-dev.service crash loop |
~123 MB/day | 13 lines / 90 s (was ~1,095) |
| Combined | ~295 MB/day | ~5 MB/day |
Netdata restarted 08:15 and reports healthy; gp-booking-dev.service disabled at 08:29. Both
changes are version-controlled in /root/work/veripath-ops (4f50d20, d47e648). The only writers
left in syslog are normal: systemd, dockerd, the capacity-collector cron and CRON.
The pass is captured as the veripath-capacity-review skill (mirrored to
the skills mirror) so the next review
starts from the ranked suspects and the safe-reclaim set rather than re-deriving them.
Trigger. Matthew: the amount being taken on is rising fast again — inevitably, because of the data feeds —
and VPS1 was seen at 60%. The question put was not "free space" but "are we managing the intake in the most
efficient way?", so this round measures the feeds separately from everything else.
Baseline. /dev/vda1 232 G, 139 G used, 93 G free (60%). Load 3.3 on 4 cores; memory 4.8 G of 16 G used.
Where it actually sits (du -x, then per-child):
| Path | Size | What it is |
|---|---|---|
/var/lib/containerd |
64 G | container image snapshots (the real image store) |
/var/lib/docker |
20 G | volumes 19 G, container layers 0.5 G |
/var/backups |
17 G | daily dumps 15 G (7-day retention), migration assets 2.2 G |
/root |
20 G | work 9.2 G, caches 3.8 G, .npm 1.4 G, .local 1.6 G, forensics 0.9 G, dd-move 0.7 G |
/usr |
5.6 G | base system |
/var/log |
1.4 G | journal 1.0 G, already capped by the round-1 drop-in |
Measured, not assumed:
snomed_relationships 606 MB + _old 559 MB, snomed_descriptions 498 MB + _old 432 MB,dmdamp 206 MB, dmdampp 195 MB, dmdamppprice 135 MB. Everything else is under 65 MB.gp_booking_pg dump wasreference-data-purge-weekly); the SNOMED loader runs DROP TABLE IF EXISTS {table}_old before renaming,retain_archive writes them to the app engine (VPS2), one per live release.gp_bookingSo the expected forward growth from the feeds is ~0.2–0.3 GB/month in the database plus a bounded 1.07 GB for the
retained previous corpus, against 93 GB free. The intake is efficient; the pressure is elsewhere.
| Item | Size | Why it is safe |
|---|---|---|
/root/work/snomed-uk-ext-20261017/extracted/ |
6.1 G | the extraction staging from the UK-extension import; regenerated from the retained archive (uk_sct2cl_43.0.0_20260923000001Z.zip, 945 MB, still in snomed-20261016/archive/). The merged build (711 MB) and the provenance record were kept |
/root/.cache/trivy |
3.0 G | scan databases, re-fetched by the weekly security scan |
| npm cache | 1.4 G | npm cache clean --force |
docker builder cache, apt lists, /tmp files >14 days |
~0.3 G | regenerable |
Result: 139 G → 129 G used, 93 G → 103 G free, 60% → 56%. All containers healthy, wiki/wiki-site/GP all 200.
nomic-embed-text so it can be recalled semanticallyintegrations/memory_service.py), and on the live dental stack on VPS2 the switch is onOLLAMA_EMBED_ENABLED=True) with a container up and serving the model. The VPS1 copy exited on 2026-09-11,11434 was sshd — a reverse SSH forward frompostgres:15 545 MB, postgres:15-alpine 408 MB, postgres:16 642 MB, pgvector:pg15 622 MB,forgejo:14.0 256 MB) and staged newer pulls of running services (netdata:latest 1.33 GB,wiki:latest 1.02 GB, keycloak:latest 752 MB, defectdojo-nginx:latest 400 MB, livekit 131 MB,client-postgres:ssh-latest 163 MB). Pruning the old variants is pure win; pruning the staged pulls means theghcr.io/zaproxy/zaproxy (3.7 GB — the monthly ZAP DAST cron/etc/cron.d/security-scanner runs it on the 1st Sunday) and the MinIO rollback image, held by theveripath-minio-rollback-keep container (protected by the 2026-10-17 ruling).Nothing in this round changed a backup, a monitoring setting or a security posture (the Ollama removal was a retired installation, authorised explicitly). The one posture-adjacent
observation: the feed growth that matters is database growth, and the daily gp_booking_pg dump size is the honest
meter for it — it moved 219 MB → 303 MB on the corpus import and has been flat since, which is what "managed" looks
like. Worth re-reading that number each quarter rather than watching df.