Last updated: 2026-09-27 — the daily freshness check was rewritten to test the dumps of record
(it had been reporting a permanent false CRITICAL); the 2026-09-26 gzip change to the database dumps in this
review. Every restore command on this page changed with it. See
Restore procedures.
Purpose: Central register of the live backup systems — locations, schedules, retention and
the restore procedure for each.
This page previously described a
/root/backup.shwriting dated directories under/root/backups/.
That script does not exist and that directory now holds incident and migration artefacts only.
The register below reflects what is actually running, verified 2026-09-26.
| Detail | Value |
|---|---|
| Script | /opt/scripts/primary-backup.sh |
| Schedule | Daily at 01:00 (from /etc/crontab) |
| Location | /var/backups/daily/ |
| Retention | 7 days, by mtime (a run or two may linger past the boundary) |
| Format | gzip — every stream. Changed 2026-09-26; previously defectdojo_pg and gp_booking_pg were plain .sql |
| Naming | <name>_<YYYYMMDD-HHMMSS>.sql.gz |
| Log | /var/log/backup-dump.log |
Sizes measured on the first compressed run, 2026-09-26 08:57:
| Stream | Covers | Compressed | Uncompressed | Ratio |
|---|---|---|---|---|
defectdojo_pg |
DefectDojo database | 827 MB | 3.30 GB | 4.3:1 |
gp_booking_pg |
whole central cluster — dsp_clinic, keycloak, wikijs, aes_portal, veripath_dental |
127 MB | 810 MB | 6.7:1 |
wikijs_pg |
Wiki.js alone (pg_dump, for a targeted restore) |
35 MB | 35 MB | already gz |
forgejo_pg |
Forgejo database | 125 KB | 891 KB | 7.1:1 |
host_pg |
host PostgreSQL (pg_dumpall) |
30 KB | 282 KB | 9.5:1 |
| Total | ≈ 1.0 GB/day | ≈ 4.4 GB/day |
Compressing the two large streams took the 7-day footprint from ≈ 33 GB to ≈ 7 GB. Compression
runs at nice -n 10 because this host also serves production all night; a full run takes ≈ 3 min.
Each stream is piped into a .tmp file, verified with gzip -t, and only then moved into place —
so a truncated dump can never be published as a good backup. set -o pipefail is set, because
previously a dump command that failed while piping into a successful gzip was logged as OK.
Verified on the first run: gzip -t clean on all five, the gp_booking CREATE DATABASE set
byte-identical to the previous uncompressed dump, the PostgreSQL database cluster dump complete
marker present, and a full restore drill passed (548 pages in production and in the restore).
| Detail | Value |
|---|---|
| Script | /opt/scripts/volume-backup.sh |
| Schedule | Daily at 01:30 (/etc/crontab) |
| Location | /var/backups/daily/volumes/ |
| Retention | 7 days |
| Contents | MinIO data/config and SeaweedFS data/config (added 2026-10-14, WO-M5 — both stores are backed up through the replacement), dental and GP media, Forgejo data, the Wiki.js content volume, Orthanc |
An encrypted Borg repository on the backup VPS, pulled nightly at 03:00. This is the
off-host copy — the dumps above live on the same host they protect. Its recency is asserted by
backup-healthcheck.sh (see Monitoring).
⚠️ Every database dump is gzipped. Decompress on the way in — the file is not readable by
psqldirectly.
Credentials are never written into this page or any command line. The central PostgreSQL password
lives in the host-local 0600 file /root/.config/veripath-db.conf; read it into the environment
the same way the backup script does:
PW=$(python3 -c "import json;print(json.load(open('/root/.config/veripath-db.conf'))['central']['password'])")
CID=gp_booking_postgres
The wikijs_pg dump is pg_dump --no-owner --no-acl, so restore it into an existing, empty
database (it contains no CREATE DATABASE):
docker exec -e PGPASSWORD="$PW" $CID dropdb -U postgres --if-exists wikijs_restore_test
docker exec -e PGPASSWORD="$PW" $CID createdb -U postgres wikijs_restore_test
zcat /var/backups/daily/wikijs_pg_<TIMESTAMP>.sql.gz \
| docker exec -i -e PGPASSWORD="$PW" $CID psql -U postgres -d wikijs_restore_test -q -v ON_ERROR_STOP=1
gp_booking_pg is a pg_dumpall, so it carries CREATE DATABASE statements and is restored into
the cluster as a whole:
zcat /var/backups/daily/gp_booking_pg_<TIMESTAMP>.sql.gz \
| docker exec -i -e PGPASSWORD="$PW" $CID psql -U postgres -v ON_ERROR_STOP=1
ls -t /var/backups/daily/gp_booking_pg_*.sql.gz | head -1
Full step-by-step, with the verification and clean-up steps, is on
Quarterly Restore Drill.
| Check | What it does | Schedule | Status |
|---|---|---|---|
/usr/local/bin/backup-healthcheck.sh |
Platform DB dump present and non-zero; Borg vault recency. Writes /tmp/platform-backup-status.json for the compliance dashboard |
Every 2 h | ✅ healthy |
/usr/local/bin/check-backup-freshness.sh |
The five database dumps of record + an integrity proof on wikijs_pg and gp_booking_pg + the volume/media archives |
Daily 07:00 | ✅ healthy (rewritten 2026-09-27 — see below) |
defectdojo_nginx_internal healthcheck |
Container health | Continuous | ✅ |
These were found while updating this page. None of them affect the dumps in section 1, which
were independently verified the same day.
Resolution, 2026-09-27 (Hermes, with Matthew's approval). Item 1 is fixed and item 2 is recorded in
the rewritten check; item 3 stands. check-backup-freshness.sh now tests what is actually the backup
of record and exits 0 on the live estate: the five dump families in /var/backups/daily with
26 h/30 h thresholds, a gzip -t integrity proof on wikijs_pg and gp_booking_pg, the volume and
media archives with a size sanity check, and the retired paths reported as information only — the
68-day-old archive it used to alert on is now printed, not treated as a failure. Its silent abort went
with it: the old script died at section 3 because set -e met an ls that matched nothing, so it never
reached its own summary. The rewrite is installed at /usr/local/bin/check-backup-freshness.sh and
versioned in veripath-ops (scripts/check-backup-freshness.sh, with the retired version kept beside it
as check-backup-freshness.sh.retired-20260927; commit d663cee).
check-backup-freshness.sh reports CRITICAL every morning./root/backups, and aborted before its summary. It now checkswikijs_pg and gp_booking_pg decompress, checks the volumevolume-backup.sh's Wiki.js content archive is a no-op — RECORDED 2026-09-27.wiki_content_<date>.tar.gz is 125 bytes and holds an empty _data/; minio_config_<date>.tar.gzwikijs_pg dump, and MinIO's objectsminio_data, so nothing is lost — the archives are now named as expected-empty in the checkEXPECTED_EMPTY) so they are reported rather than alerted on, and a size drop in any other archivebackup-healthcheck.sh checks size, not integrity. It asserts a dump is non-zero but doesgzip -t.