Purpose: Prove the backups are actually restorable, not merely present. Two drills on this page:
Drill A — the database dumps (restore the newest dump into a scratch database, compare with production) and
Drill B — the imaging store (restore the SeaweedFS radiograph volume into a scratch tree, serve it, and
compare it object-for-object with the live store).
Frequency: Quarterly for both; Drill B also immediately before any decommission decision — it is the
restore-drill half of the WO-M6 gate on the MinIO → SeaweedFS plan.
Owner: Infrastructure team
Last run: 2026-10-15 — PASS, Drill B (imaging store); Drill A last ran 2026-09-26 — PASS (see Drill History)
⚠️ All dumps are gzipped. Decompress on the way in —
zcat, orpsqlwill choke on the
binary. The dump filename ends.sql.gz.This page previously carried the central database password as a literal in every command, and
referenced the pre-rename container3ff9c011e629_gp_booking_postgres. Both are fixed here.
Credentials must never be written into this page — read them from the host-local0600config
as shown below.
Confirm the containers are up and read the credential into the environment. The password is never
echoed and never typed into a command literal — it is read from the same host-local 0600 file
the backup script uses:
docker ps --filter name=gp_booking_postgres --format '{{.Names}} {{.Status}}'
docker ps --filter name=wikijs --format '{{.Names}} {{.Status}}'
PW=$(python3 -c "import json;print(json.load(open('/root/.config/veripath-db.conf'))['central']['password'])")
CID=gp_booking_postgres
# Pick the newest Wiki.js dump (never hard-code a timestamp)
DUMP=$(ls -t /var/backups/daily/wikijs_pg_*.sql.gz | head -1)
echo "Restoring from: $DUMP"
# A fresh, clearly-named scratch database
docker exec -e PGPASSWORD="$PW" $CID dropdb -U postgres --if-exists wikijs_restore_test
docker exec -e PGPASSWORD="$PW" $CID createdb -U postgres wikijs_restore_test
# Decompress on the way in. ON_ERROR_STOP makes a partial restore fail loudly.
zcat "$DUMP" | docker exec -i -e PGPASSWORD="$PW" $CID \
psql -U postgres -d wikijs_restore_test -q -v ON_ERROR_STOP=1
echo "psql exit: $? (0 and no stderr output = clean restore)"
PROD=$(docker exec -e PGPASSWORD="$PW" $CID \
psql -U postgres -d wikijs -t -A -c "SELECT COUNT(*) FROM pages;")
TEST=$(docker exec -e PGPASSWORD="$PW" $CID \
psql -U postgres -d wikijs_restore_test -t -A -c "SELECT COUNT(*) FROM pages;")
echo "Production pages: $PROD"
echo "Restored pages: $TEST"
[ "$PROD" = "$TEST" ] && echo "PASS" || echo "FAIL — page count mismatch"
# Spot-check that real content came back, not just a schema
docker exec -e PGPASSWORD="$PW" $CID \
psql -U postgres -d wikijs_restore_test -t -A -c "SELECT title FROM pages ORDER BY id LIMIT 3;"
Expect a match — production moves between the dump and the drill, so a small drift is normal if
a long time has passed; a large or structural difference is not.
The scratch database must not be left behind:
docker exec -e PGPASSWORD="$PW" $CID dropdb -U postgres --if-exists wikijs_restore_test
# Confirm nothing was left over
docker exec -e PGPASSWORD="$PW" $CID psql -U postgres -t -A \
-c "SELECT datname FROM pg_database WHERE datname LIKE '%restore%';"
Add a row to the Drill History table below.
The radiograph volume is not a database: it is a directory of SeaweedFS volume files, so "restorable" means the
restored copy starts and serves the same objects. Both the primary's /opt/seaweedfs/data and its
s3.json credential are captured daily by volume-backup.sh into
/var/backups/daily/volumes/, then pulled into the offsite vault.
TAR=$(ls -t /var/backups/daily/volumes/seaweedfs_data_*.tar.gz | head -1) # never hard-code a timestamp
gzip -t "$TAR" && echo "archive intact"
S=/var/tmp/restore-drill-$(date +%Y%m%d-%H%M%S); mkdir -p "$S/restored" "$S/restored-config"
tar -xzf "$TAR" -C "$S/restored" # extracts as restored/data/
tar -xzf "$(ls -t /var/backups/daily/volumes/seaweedfs_config_*.tar.gz | head -1)" -C "$S/restored-config"
The config tar carries s3.json at 0600 — the credential travels with the data, as volume-backup.sh
intends. Confirm the vault holds the same tar (the backup VPS, described by name — no addresses in this page):
ssh root@<backup-vps> 'export BORG_PASSPHRASE=$(grep -oP "(?<=BORG_PASSPHRASE=).*" /opt/scripts/backup-pull.sh | head -1);
LAST=$(borg list --last 1 /var/backups/clinic_vault | head -1 | awk "{print \$1}");
borg list /var/backups/clinic_vault::$LAST | grep -i seaweedfs'
Never point this at the live path, and never publish it beyond the host:
docker run -d --name restore-drill-weed \
-p 127.0.0.1:19000:9000 -p 127.0.0.1:18888:8888 \
-v "$S/restored/data":/data -v "$S/restored-config/s3.json":/s3.json:ro \
chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d \
server -s3 -s3.port=9000 -s3.config=/s3.json -dir=/data
# readiness: poll, don't sleep — and check the code, not the loop's exit status
for i in $(seq 1 40); do
c=$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 http://127.0.0.1:18888/)
[ "$c" = "200" ] && { echo "filer ready after ${i}s"; break; }; sleep 1
done
Read the credential from the host-local file and list both endpoints with boto3 — bucket names, object
counts, total bytes and the md5 of a few samples:
IP=$(docker inspect seaweedfs --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}')
python3 - "$IP" <<'PY'
import json, sys, hashlib, boto3
from botocore.config import Config
creds = json.load(open('/opt/seaweedfs/s3.json'))['identities'][0]['credentials'][0]
def survey(ep):
c = boto3.client('s3', endpoint_url=ep, aws_access_key_id=creds['accessKey'],
aws_secret_access_key=creds['secretKey'], region_name='us-east-1',
config=Config(s3={'addressing_style': 'path'}))
for b in sorted(x['Name'] for x in c.list_buckets()['Buckets']):
n = t = 0; samples = []
for page in c.get_paginator('list_objects_v2').paginate(Bucket=b):
for o in page.get('Contents', []):
n += 1; t += o['Size']
if len(samples) < 3:
body = c.get_object(Bucket=b, Key=o['Key'])['Body'].read()
samples.append((o['Key'].split('/')[-1], len(body), hashlib.md5(body).hexdigest()))
print(f" {b}: {n} objects, {t} bytes"); [print(" ", s) for s in samples]
print("LIVE:"); survey(f'http://{sys.argv[1]}:9000')
print("RESTORED:"); survey('http://127.0.0.1:19000')
PY
Expect identical on all four measures. A mismatch in counts or bytes is a failed drill, not drift — the
store is append-mostly, so anything written since the backup shows up as extra objects in the live listing.
docker rm -f restore-drill-weed
rm -rf "$S"
docker ps -a --format '{{.Names}}' | grep -i drill || echo "no drill containers left"
docker inspect seaweedfs --format 'StartedAt={{.State.StartedAt}} Restarts={{.RestartCount}} Health={{.State.Health.Status}}'
The live container must show the same StartedAt, Restarts=0 and a healthy status as before the drill,
and the live directory the same file count and byte total. Then re-list the live bucket to confirm it still
serves.
Add a row to the Drill History table below.
| Date | Result | Notes | Verified By |
|---|---|---|---|
| 2026-09-26 | PASS | Ran as part of the dump-compression change, to prove the new gzipped path restores. Source wikijs_pg_20260926-085702.sql.gz (35 MB). Restore clean: psql exit 0, zero stderr, ON_ERROR_STOP set. 548 pages production / 548 restored — match. Spot-check returned real page titles. Scratch database dropped and confirmed gone; production untouched. |
Hermes Agent |
| 2026-10-15 | PASS | Drill B (imaging store) — the first run, and it is the restore-drill half of the WO-M6 gate. Source seaweedfs_data_20261015-013001.tar.gz (17,151,649 bytes, taken 01:30 after the SeaweedFS swap). gzip -t clean, extract clean into a scratch tree: 86 files / 18,467,929 bytes — identical to the live directory, zero size differences across all entries. The restored copy was served on a throwaway container (loopback only) and started cleanly; comparing both stores over S3 with boto3: same bucket, 67 objects, 18,389,458 bytes, three sample PNGs md5-identical. The 0600 s3.json restored from its own tar, and the offsite vault (backup-20261015-030126) was confirmed to hold the same tar. Throwaway container removed, scratch tree deleted, no leftovers; live store untouched — same StartedAt, 0 restarts, healthy, and still serving 67 objects / 18,389,458 bytes after teardown. |
Hermes Agent |
Not yet covered by a drill: a whole-cluster restore from gp_booking_pg_*.sql.gz (a
pg_dumpall carrying CREATE DATABASE). That is a destructive, whole-instance operation and needs
a planned window — it should be its own scheduled exercise, not part of this drill.
Also not covered: the dental Client VPS's own store (which also serves the pharmacy tenant on that VPS). The
primary's store is covered by Drill B; this one is not — but not because we cannot reach it: we hold root SSH to
it (verified 2026-10-15, see the WO-M4 runbook),
so the obstacle is authorisation and a decision, not access. Updated 2026-10-15: that box's store is now
SeaweedFS as well (WO-M4 swapped it in place at port parity; MinIO stopped and retained for rollback), so
drilling its store is Drill B pointed at that box rather than a new procedure — and the accepted risk for the
store is closed on both copies. What is still open is the agent half: with no patch agent installed there,
that box's image_refresh list and the maintenance loop it belongs to cannot be exercised yet.