| Key | Value |
|---|---|
| name | veripath-capacity-review |
| description | The recurring VPS1 capacity and tidy-up pass — measure-first method, the ranked growth suspects and their commands, the known drain signatures (Netdata/AppArmor audit flood, crash-looping systemd units, uncompressed DB dumps, uncapped journal), the safe regenerable reclaim set, the never-touch list, and where the round is recorded. Load whenever capacity is questioned, free space is "diminishing", or a general system tidy-up is asked for. |
| version | 1.1.0 |
| author | Hermes Agent |
| license | MIT |
| platforms | [linux] |
| metadata | |
| hermes | |
| tags | [veripath, capacity, disk, docker, logs, journald, apparmor, netdata, systemd, backups, housekeeping, ops] |
| category | veripath |
| related_skills | [veripath-client-vps-ops, veripath-wiki-housekeeping, veripath-build-verification] |
veripath-wiki-housekeeping, which handles the wiki; this one handles the host).Record every round on the wiki page infrastructure/disk-capacity-review (living page, newest
round appended, dated). The reply to Matthew stays short — the page carries the detail.
df -h / alone tells you nothing. Establish the rate, not just the total:
df -h / ; df -i / # space and inodes
du -xhd1 / 2>/dev/null | sort -h # top-level: /var is almost always the story
a=$(stat -c%s /var/log/syslog); sleep 20; b=$(stat -c%s /var/log/syslog)
echo "syslog +$((b-a)) bytes/20s" # per-second growth rate of a suspect log
Then work the suspects below in order. -xdev/-x matters: without it du walks into the Docker
overlays and reports the same bytes many times.
Check the log facility for one endlessly repeating line, not many different ones:
grep -c 'type=1400' /var/log/syslog # AppArmor denials
tail -c 200M /var/log/syslog | sed 's/^[A-Z][a-z][a-z] [ 0-9][0-9] [0-9:]* //' \
| cut -c1-90 | sort | uniq -c | sort -rn | head
Signature seen 2026-09-26: apparmor="DENIED" operation="ptrace" profile="docker-default" comm="apps.plugin" — Netdata's per-process plugin retrying a ptrace its container profile denies,
forever. Baseline measured 158,400 lines/day, 172 MB/day into syslog alone, plus ~45 MB/day in
kern.log and the journal. It is denied, so it collects nothing — the noise is pure loss.
Fix applied 2026-09-26: [plugins] apps = no in /opt/infra/netdata/netdata.conf, container
restarted. Verified 0 new denials in a 120s window afterwards. Costs only the per-process
"Applications" charts (never populated from the denied ptrace anyway). The
security_opt: [apparmor=unconfined] alternative was rejected — it keeps the charts but weakens
isolation on a container that already holds the Docker socket and runs on the host network.
Mirror the config into /root/work/veripath-ops/infra/netdata/ and commit; it is version-controlled.
After fixing a flood, re-sample syslog: whatever is still writing is usually a different single
source. On 2026-09-26 that was ~126 MB/day from a unit that could not boot.
tail -300 /var/log/syslog | sed 's/^[A-Za-z0-9:+.T-]* //' | cut -d' ' -f1-2 \
| sort | uniq -c | sort -rn | head -5 # who is actually writing
systemctl list-units --state=activating,failed --no-pager
systemctl show <unit> -p NRestarts -p Restart -p RestartSec
ss -ltnp | grep <port> # prove it provides NOTHING
Tells: repeated Started X / Failed with result 'exit-code' / Scheduled restart job, restart counter is at NNNNN, and a huge NRestarts.
The rate-limiter trap: systemd's default StartLimitBurst=5 over StartLimitIntervalSec=10s
can never trip when RestartSec is ≥10s — so the unit loops forever instead of being
rate-limited into failed. The observed unit had 195,893 restarts and 2.2M journal lines.
Diagnose with journalctl -u <unit> -n 40 (the traceback names the cause; here
ModuleNotFoundError: No module named 'channels').
⚠️ A unit is a decision, not a cache — report it, do not unilaterally disable it. It may be
someone's dev environment. Check the estate docs first (VPS1 is documented as running no GP booking
runtime; the app lives on the app engine).
du -xhd1 /var/backups | sort -h ; ls -lS /var/backups/daily | head -30
/opt/scripts/primary-backup.sh retains 7 days (-mtime +7 -delete), so the footprint
plateaus — it is a churn, not an unbounded leak. But two streams are written uncompressed
(defectdojo_pg ≈3.55 GB/day, gp_booking_pg ≈0.85 GB/day ≈ 4.4 GB/day written) while the
same script already gzips the wikijs dump. ⚠️ Compressing them changes the restore path —
update infrastructure/backups/overview and the quarterly drill in the same change, and commit to
/root/work/veripath-ops (version-controlled ops).
journalctl --disk-usage ; grep -vE '^\s*#|^\s*$' /etc/systemd/journald.conf
With SystemMaxUse unset the ceiling is 10% of the filesystem (≈23G here). Cap it:
drop-in /etc/systemd/journald.conf.d/99-veripath-cap.conf with SystemMaxUse=1G, then
systemctl restart systemd-journald and confirm with systemd-analyze cat-config systemd/journald.conf.
Vacuuming discards history — say so on the record, and point at the rotated syslog.*.gz as the
surviving deeper history.
A weekly rotation with no size limit lets one file hit a gigabyte. Add maxsize 100M to the
rsyslog stanza (/etc/logrotate.d/rsyslog; keep a .bak-<date>). Retention is unchanged — this
only bounds how big one file may grow, so no source can ever blow the disk again.
Also: /var/log/dmesg is matched by no rotation rule — truncate it, it is rewritten at boot.
docker system df ; docker system df -v | sed -n '/VOLUME NAME/,$p'
docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Size}}'
Images ... RECLAIMABLE is misleading: with zero dangling images it is just images referenced
only by stopped containers. The real prize is usually a single stopped container's writable
layer. On 2026-09-26 local_hermes_agent alone held 28.7 GB. Always inspect before removing.
/root/.cache/trivy (scan DBs, re-fetched by the weekly scan) · /tmp/trivy-scans · npm
_cacache for root and matthew (NOT .npm-global, that is installed packages) ·
docker builder prune -af · apt-get clean · /tmp scratch files older than 14 days.
journalctl --vacuum-size=1G
cp -a /etc/logrotate.d/rsyslog /etc/logrotate.d/rsyslog.bak-$(date +%Y%m%d)
# ...install maxsize 100M... then rotate TWICE, so delaycompress completes:
logrotate -f /etc/logrotate.d/rsyslog && sleep 2 && logrotate -f /etc/logrotate.d/rsyslog
: > /var/log/dmesg
rm -rf /root/.cache/trivy/* /tmp/trivy-scans/*
docker builder prune -af
npm cache clean --force
apt-get clean
find /tmp -maxdepth 1 -mtime +14 -type f -delete
Measured on the 2026-09-26 pass: 12 GB (149G → 137G used, 65% → 59%).
Never reclaim without an explicit decision (assets, evidence, or another host's rollback):
defectdojo_stage_defectdojo_postgres volume (~4.6 GB) — the documented pre-move DefectDojo/root/forensics — incident evidence. /root/dd-move, /var/backups/migration-images,/var/backups/premigration — migration assets: superseded-looking but ask.docker compose down or docker system prune -a --volumes --force —/opt/scripts/platform-vps-health.sh collects disk_usage_pct every 5 min into the status JSON
synced to the app engine. Disk alerting exists; a capacity pass restores headroom, it does not add
the watchdog.
After every pass: docker ps (no unhealthy/restarting), systemctl is-active systemd-journald rsyslog docker nginx, curl the wiki and the public site for 200, then record
df -h / before/after on infrastructure/disk-capacity-review and confirm the rendered page
(anchor-check a few strings — a stale SSR cache serves the old body).
Prove a fix with a before/after count over a fixed window (grep -c twice around a sleep),
never with a single reading.
du without -x double-counts Docker overlay paths.logrotate -f once is not enough when delaycompress is set — the big file stays uncompressedgrep -c '[a]pps.plugin' counts your own command line — use a bracketed pattern and read thedescription column is varchar(255) — a longer one fails the whole upsert./opt/hermes-skills-sync.py carries a hardcoded VERIPATH_SKILLS list: a new skill does not