Backup & Restore¶

Overview¶

Heezy uses two backup tiers:

Tier	Schedule	Destination	Retention
Local NFS	Daily 02:00 UTC	`big-boi:/nfs/heezy/backups/`	7 days
Off site S3	Saturday 08:00 UTC	`s3://heezy-backups-025066240222/heezy/`	30 days, STANDARD_IA

Two Ansible plays, heezy-backup and heezy-backup-offsite. Prefer the NFS copies for restores: they hold 7 daily generations and are immediately accessible.

S3 holds four weekly generations, not thirty daily ones. The off-site run ships one file per series, so if you need a specific day older than the local 7 day window and it was not a Saturday, it does not exist. Restructured 2026-08-19; see Backups.

Covered Services¶

All services in the heezy namespace are backed up daily:

Service	Container Path	Notes
overseerr	`/config/db`	Full `/config` causes `initialized: false` on restore
radarr	`/config`
sonarr	`/config`
lidarr	`/config`
prowlarr	`/config`
sabnzbd	`/config`
qbittorrent	`/config`	selector: `app=qbittorrent-vpn`, container: `qbittorrent`
tautulli	`/config`
navidrome	`/data`
pihole	`/etc/pihole`
uptime-kuma	`/app/data`
plex	`Preferences.xml`, `Plug-in Support/Databases`, `Plug-in Support/Metadata Combination`	Not full `/config`. `Media`, `Metadata` and `Cache` are regenerable and Plex rebuilds them. Restore into `/config/Library/Application Support/Plex Media Server/`
lazylibrarian	`/config`
aurral	`/app/backend/data`
swag	`/config`
slskd	`/app`	selector: `app=slskd-vpn`, container: `slskd`, excludes downloads
tailscale	`/var/lib/tailscale`
emulatorjs	`/config`
heezy-finance	`/data`
openbao	raft snapshot	namespace: `openbao`

Also backed up:

What	Destination	Notes
Postgres `heezy` DB	`/nfs/heezy/backups/postgres/heezy/`	pg_dump via SSH to big-boi
Postgres `n8n` DB	`/nfs/heezy/backups/n8n/`	`docker exec n8n-postgres pg_dump` via SSH
n8n data dir	`/nfs/heezy/backups/n8n/`	tar of `/opt/big-boi-ai/n8n/`
open-webui	`/nfs/heezy/backups/open-webui/`	tar of `/opt/big-boi/open-webui/` (excludes cache)
Game servers	`/nfs/heezy/backups/games/`	Six archives, read out of each pod. See Game Server On Macvlan
Network configs	`/nfs/heezy/backups/network/`	FortiGate REST, Cisco sshpass, Ruckus session cookie

Restore from NFS (preferred)¶

NFS backups are at big-boi:/nfs/heezy/backups/. No retrieval wait.

Postgres (any DB)¶

ls -lt /nfs/heezy/backups/postgres/heezy/

gunzip -c /nfs/heezy/backups/postgres/heezy/heezy-YYYYMMDD_HHMMSS.sql.gz \
  | PGPASSWORD=<password> psql -h postgres.heezy.local -U heezy_app heezy

OpenBao Raft snapshot¶

Use the dedicated runbook: Restore OpenBao From a Raft Snapshot.

The two commands below only work when OpenBao is already initialized and unsealed, which is the easy case. The case you will actually hit is an uninitialized instance, where snapshot restore has nothing to authenticate against and this fails. There is also a token reviewer JWT that has to be replaced afterwards or every ExternalSecret returns 403. Both are covered in the runbook, which was walked end to end against real data on 2026-08-16.

ls -lt /nfs/heezy/backups/k8s/openbao/

sudo microk8s kubectl cp \
  /nfs/heezy/backups/k8s/openbao/openbao-TIMESTAMP.snap \
  openbao/openbao-0:/tmp/openbao.snap

sudo microk8s kubectl exec -n openbao openbao-0 -- \
  vault operator raft snapshot restore /tmp/openbao.snap

K8s PVC volume¶

Full step-by-step process:

1. Scale down the service:

sudo microk8s kubectl scale deployment/<service> -n heezy --replicas=0

2. Find which node the PVC is bound to:

sudo microk8s kubectl get pods -n heezy -l app=<service> -o wide
# Note the NODE column before scaling down, or check:
sudo microk8s kubectl get pvc -n heezy | grep <service>

3. Spin up a restore pod (use base64 to avoid SSH quoting issues):

Generate the manifest and base64-encode it locally:

cat <<'MANIFEST' | base64 -w0
apiVersion: v1
kind: Pod
metadata:
  name: <service>-restore
  namespace: heezy
spec:
  nodeSelector:
    kubernetes.io/hostname: <node-where-pvc-lives>
  restartPolicy: Never
  volumes:
  - name: config
    persistentVolumeClaim:
      claimName: <service>-config
  containers:
  - name: restore
    image: alpine
    command: ["sleep", "3600"]
    volumeMounts:
    - name: config
      mountPath: /config
MANIFEST

Apply on the node:

echo <base64> | base64 -d > /tmp/restore-pod.yaml
sudo microk8s kubectl apply -f /tmp/restore-pod.yaml
sudo microk8s kubectl wait pod <service>-restore -n heezy --for=condition=Ready --timeout=60s

4. Download backup from NFS, copy into the pod:

# SCP from big-boi to the nebula node
scp mcp-admin@bigboi.heezy.local:/nfs/heezy/backups/k8s/<service>/<service>-TIMESTAMP.tar.gz /tmp/

# Copy into the restore pod
sudo microk8s kubectl cp /tmp/<service>-TIMESTAMP.tar.gz heezy/<service>-restore:/tmp/backup.tar.gz

5. Extract into the PVC:

sudo microk8s kubectl exec -n heezy <service>-restore -- sh -c "
  cp -r /config /config.bak.$(date +%Y%m%d) &&
  tar -xzf /tmp/backup.tar.gz -C /config --strip-components=1
"

For SQLite databases (overseerr, etc.) — remove WAL/SHM before scaling up:

sudo microk8s kubectl exec -n heezy <service>-restore -- sh -c "
  rm -f /config/db/db.sqlite3-shm /config/db/db.sqlite3-wal
"

6. Delete restore pod and scale back up:

sudo microk8s kubectl delete pod <service>-restore -n heezy
sudo microk8s kubectl scale deployment/<service> -n heezy --replicas=1
sudo microk8s kubectl wait --for=condition=ready pod -l app=<service> -n heezy --timeout=60s
sudo microk8s kubectl logs -n heezy -l app=<service> --tail=20

Overseerr — `initialized: false` fix¶

If overseerr shows the setup wizard after restore, settings.json has initialized: false. Fix via the restore pod before scaling up:

# Copy settings.json out
sudo microk8s kubectl cp heezy/overseerr-restore:/config/settings.json /tmp/overseerr-settings.json
scp mcp-admin@nebula1.heezy.local:/tmp/overseerr-settings.json /tmp/

# Fix the flag
python3 -c "
import json
with open('/tmp/overseerr-settings.json') as f: s = json.load(f)
s['public']['initialized'] = True
with open('/tmp/overseerr-settings-fixed.json', 'w') as f: json.dump(s, f, indent=1)
"

# Push back
scp /tmp/overseerr-settings-fixed.json mcp-admin@nebula1.heezy.local:/tmp/
sudo microk8s kubectl cp /tmp/overseerr-settings-fixed.json heezy/overseerr-restore:/config/settings.json

Then delete the restore pod and scale back up.

Docker Compose (big-boi)¶

n8n data dir¶

ssh mcp-admin@bigboi.heezy.local "cd /opt/big-boi-ai && sudo docker compose stop n8n"
ssh mcp-admin@bigboi.heezy.local "sudo tar -xzf /nfs/heezy/backups/n8n/n8n-data-TIMESTAMP.tar.gz -C /opt/big-boi-ai/n8n/"
ssh mcp-admin@bigboi.heezy.local "cd /opt/big-boi-ai && sudo docker compose up -d n8n"

n8n postgres¶

ssh mcp-admin@bigboi.heezy.local "cd /opt/big-boi-ai && sudo docker compose stop n8n"
ssh mcp-admin@bigboi.heezy.local "gunzip -c /nfs/heezy/backups/n8n/n8n-postgres-TIMESTAMP.sql.gz | sudo docker exec -i n8n-postgres psql -U n8n n8n"
ssh mcp-admin@bigboi.heezy.local "cd /opt/big-boi-ai && sudo docker compose up -d n8n"

open-webui¶

ssh mcp-admin@bigboi.heezy.local "cd /opt/big-boi && sudo docker compose stop open-webui"
ssh mcp-admin@bigboi.heezy.local "sudo tar -xzf /nfs/heezy/backups/open-webui/open-webui-TIMESTAMP.tar.gz -C /opt/big-boi/open-webui/"
ssh mcp-admin@bigboi.heezy.local "cd /opt/big-boi && sudo docker compose up -d open-webui"

Restore from S3¶

Use this when NFS backups have aged out (beyond 7 day retention) or NFS is unavailable.

No retrieval wait, as of 2026-08-19

Objects are written as STANDARD_IA and are immediately downloadable. No restore-object request, no thaw. This changed with the 30 day retention: GLACIER Flexible Retrieval bills a 90 day minimum per object, so a 30 day GLACIER tree costs the same as STANDARD_IA while taking 3 to 5 hours to read.

Objects written before 2026-08-19 are still GLACIER or DEEP_ARCHIVE and do need a thaw. The 30 day lifecycle expires them, so they will be gone shortly. Use the restore-object form below only if you hit one.

# Confirm the storage class before assuming
aws s3api list-objects-v2 --bucket heezy-backups-025066240222 \
  --prefix heezy/k8s/<service>/ \
  --query "Contents[*].{Key:Key,StorageClass:StorageClass}" \
  --output table | tail -10

# STANDARD_IA — just download
aws s3 cp s3://heezy-backups-025066240222/heezy/k8s/<service>/<backup>.tar.gz /tmp/

# Only for pre-2026-08-19 GLACIER or DEEP_ARCHIVE objects. Standard tier is 3-5 hours
# for GLACIER, 12 hours for DEEP_ARCHIVE. Expedited is 1-5 minutes, GLACIER only.
aws s3api restore-object \
  --bucket heezy-backups-025066240222 \
  --key heezy/k8s/<service>/<backup>.tar.gz \
  --restore-request '{"Days":3,"GlacierJobParameters":{"Tier":"Standard"}}'

aws s3api head-object \
  --bucket heezy-backups-025066240222 \
  --key heezy/k8s/<service>/<backup>.tar.gz \
  --query "Restore"
# Ready when: ongoing-request="false"

From S3, download to MCP container first (nebula nodes don't have AWS credentials), then SCP to a node, then kubectl cp into the restore pod:

# On MCP container
aws s3 cp s3://heezy-backups-025066240222/heezy/k8s/<service>/<backup>.tar.gz /tmp/
scp -i /root/.ssh/mcp_heezy /tmp/<backup>.tar.gz mcp-admin@nebula1.heezy.local:/tmp/

# On nebula node
sudo microk8s kubectl cp /tmp/<backup>.tar.gz heezy/<service>-restore:/tmp/backup.tar.gz

Legacy S3 backup format (pre-July 2026 wipe)¶

Backups created before the July 2026 data wipe used a different format: the tarball contains a single .db file (e.g. radarr-20260612_101851.db) rather than a full /config directory tree. This affects radarr, sonarr, lidarr, and prowlarr backups from before that date.

Extraction for these old backups differs from the standard procedure:

# Extract to /tmp first — do NOT extract directly to /config
sudo microk8s kubectl exec -n heezy <service>-restore -- tar -xzf /tmp/backup.tar.gz -C /tmp/

# The extracted file will be named like <service>-TIMESTAMP.db — rename it
sudo microk8s kubectl exec -n heezy <service>-restore -- \
  sh -c 'mv /tmp/<service>-*.db /config/<service>.db'

# Remove any stale WAL/SHM
sudo microk8s kubectl exec -n heezy <service>-restore -- \
  sh -c 'rm -f /config/<service>.db-shm /config/<service>.db-wal'

All backups created by the current heezy-backup Ansible role use the full /config tar format and follow the standard restore procedure above.

Prowlarr — credential reset¶

If Prowlarr credentials are lost, reset via direct SQLite edit. There is no sqlite3 binary in the container — use a restore pod on nebula-4 (where prowlarr runs).

Prowlarr uses PBKDF2-HMAC-SHA1, 10000 iterations, 32-byte derived key, base64-encoded salt and hash stored in the Users table.

# Scale down
ssh mcp-admin@nebula1.heezy.local 'sudo microk8s kubectl scale deployment prowlarr -n heezy --replicas=0'

# Spin up alpine pod with PVC mounted — base64 the manifest to avoid quoting issues
cat <<'EOF' | base64 -w0
apiVersion: v1
kind: Pod
metadata:
  name: prowlarr-reset
  namespace: heezy
spec:
  nodeSelector:
    kubernetes.io/hostname: nebula-4
  restartPolicy: Never
  volumes:
  - name: config
    persistentVolumeClaim:
      claimName: prowlarr-config
  containers:
  - name: reset
    image: alpine
    command: ["sleep", "600"]
    volumeMounts:
    - name: config
      mountPath: /config
EOF
# Pipe the above to: ssh mcp-admin@nebula1.heezy.local 'base64 -d > /tmp/p.yaml && sudo microk8s kubectl apply -f /tmp/p.yaml'

# Install sqlite in the pod
ssh mcp-admin@nebula1.heezy.local 'sudo microk8s kubectl exec -n heezy prowlarr-reset -- apk add --quiet sqlite'

# Generate hash and update — run this python3 script on the MCP container or nebula-1
# (uses subprocess to call kubectl exec, avoiding all quoting issues)
python3 - <<'PYEOF'
import subprocess, hashlib, base64, os
password = "your-new-password"
iterations = 10000
salt = os.urandom(16)
dk = hashlib.pbkdf2_hmac('sha1', password.encode('utf-8'), salt, iterations, dklen=32)
salt_b64 = base64.b64encode(salt).decode()
hash_b64 = base64.b64encode(dk).decode()
sql = f"UPDATE Users SET Username='admin', Password='{hash_b64}', Salt='{salt_b64}', Iterations={iterations} WHERE Id=1;"
r = subprocess.run(
    ["sudo","microk8s","kubectl","exec","-n","heezy","prowlarr-reset","--",
     "sqlite3","/config/prowlarr.db", sql],
    capture_output=True, text=True)
print("RC:", r.returncode, r.stderr or "OK")
PYEOF

# Clean up and scale back up
ssh mcp-admin@nebula1.heezy.local 'sudo microk8s kubectl delete pod prowlarr-reset -n heezy'
ssh mcp-admin@nebula1.heezy.local 'sudo microk8s kubectl scale deployment prowlarr -n heezy --replicas=1'

Key detail: dklen=32 is required. Omitting it produces a 20-byte key that won't match what Prowlarr expects.

Manual backup trigger¶

Via Gitea Actions:

Gitea → heezy-admin/ansible-heezy → Actions → playbook-heezy-backup-execution.yml → Run workflow (main branch).

Via Gitea API:

curl -sk -X POST "http://gitea.heezy.local:3000/api/v1/repos/heezy-admin/ansible-heezy/actions/workflows/playbook-heezy-backup-execution.yml/dispatches" \
  -u "heezy-admin:HeezyGit2026!" \
  -H "Content-Type: application/json" \
  -d '{"ref":"main"}'

Verify backup health¶

# Recent backups on NFS (files modified in the last 2 days)
ssh mcp-admin@bigboi.heezy.local "find /nfs/heezy/backups -name '*.tar.gz' -mtime -2 | sort"
ssh mcp-admin@bigboi.heezy.local "find /nfs/heezy/backups -name '*.snap' -mtime -2 | sort"

# List k8s backup dirs
ssh mcp-admin@bigboi.heezy.local "ls /nfs/heezy/backups/k8s/"

# S3 listing
aws s3 ls s3://heezy-backups-025066240222/heezy/ --recursive --human-readable | sort | tail -20

Known gaps¶

TrueNAS (nfs.heezy.local) has no backup coverage. A NAS failure loses every local copy, leaving only the last weekly S3 generation — up to 7 days of exposure since the off-site ship moved to weekly on 2026-08-19.
LGTM metrics (Mimir/Prometheus data, ~115 GB) are not backed up. Only configs are preserved. Metrics will regenerate from scrapers after a restore.
qbittorrent has no S3 backup history prior to the July 2026 wipe. First usable backups will be from the first successful heezy-backup run post-wipe. (nzbhydra2 was decommissioned 2026-08-08.)
Off-site granularity is weekly, not daily. Local NFS holds 7 daily generations; S3 holds 4 weekly ones. If you need a specific day older than the local window and it was not a Saturday, it does not exist. See Backups.
SABnzbd S3 backup was confirmed corrupt (45 bytes) prior to the July 2026 wipe. Usenet server credentials and categories must be reconfigured manually after a restore — they are not recoverable from backup.