Restore OpenBao From a Raft Snapshot¶

Full recovery of OpenBao when its storage is empty, corrupt, or rolled back to a bad state. Written to be followed start to finish without prior OpenBao knowledge.

Read this first

Do not run bao operator init on an OpenBao that still has its data. Init is only correct when the instance reports Initialized false. Check with Step 2 before touching anything.

Do not overwrite the openbao-init-keys secret at any point. The unseal keys it holds belong to the barrier inside the snapshot and become valid again once the restore finishes. Losing that secret means losing OpenBao permanently.

Last performed: 2026-08-16, successfully, on the real cluster. Notes from that run are folded into the steps below.

When to use this¶

Symptom	This runbook?
`bao status` says `Initialized false`	Yes
Pod logs repeat `security barrier not initialized`	Yes
Secrets return 404 that used to exist	Yes
`bao status` says `Sealed true` but `Initialized true`	No. It is just sealed. See Only sealed
ExternalSecrets failing but OpenBao is unsealed	No. Skip to Step 8

What you are restoring from¶

The heezy-backup Ansible role backs OpenBao up as a Raft snapshot, not a volume tar. The target is declared in ansible-heezy/roles/heezy-backup/defaults/main.yml:

- { name: openbao, namespace: openbao, label_selector: "app.kubernetes.io/name=openbao", type: raft-snapshot }

It calls OpenBao's own snapshot API with the root token, so the file is a consistent point-in-time copy of the entire barrier: every secret, auth method, policy, and AppRole.

Where	Path	Cadence	Retention
NFS (use this)	`big-boi:/nfs/heezy/backups/k8s/openbao/openbao-YYYYMMDD_HHMMSS.snap`	daily 02:00 UTC	7 days
S3	`s3://heezy-backups-025066240222/heezy/k8s/openbao/`	monthly, 1st 03:00 UTC	STANDARD_IA, expires at 90 days

Snapshots are small, roughly 100KB to 150KB. A file much smaller than that is suspect.

What you need before starting¶

kubectl against the cluster, from the MCP container or a node
SSH to big-boi as mcp-admin (key /root/.ssh/mcp_heezy in the MCP container)
Nothing else. The unseal keys and root token are already in the cluster

Step 1: Confirm the pod is running¶

kubectl get pods -n openbao -l app.kubernetes.io/name=openbao -o wide

You want openbao-0 in Running, even if it shows 0/1. 0/1 is expected and just means the readiness probe fails because OpenBao is sealed.

If it is Pending with a Multi-Attach error, the Longhorn volume is still attached to a node that has not released it. Wait. It clears on its own within about a minute.

Step 2: Decide which situation you are in¶

kubectl exec -n openbao openbao-0 -- sh -c "BAO_ADDR=http://127.0.0.1:8200 bao status"

Read two lines of the output:

Initialized        false     <- storage is empty, continue with this runbook
Sealed             true

Initialized false means the Raft store has no barrier. Continue to Step 3.
Initialized true and Sealed true means the data is fine and it only needs unsealing. Jump to Only sealed.
Initialized true and Sealed false means OpenBao is healthy. You probably want Step 8 instead.

Step 3: Stop the unseal CronJob¶

A CronJob named openbao-unseal runs every 5 minutes and tries to unseal using the keys in openbao-init-keys. During this procedure the instance briefly has different temporary keys, and letting the CronJob fire against it mid-restore causes confusing failures.

kubectl patch cronjob -n openbao openbao-unseal -p '{"spec":{"suspend":true}}'

Do not forget to undo this

Step 9 turns it back on. If you leave it suspended, OpenBao will not auto recover the next time it restarts.

Step 4: Pick and fetch a snapshot¶

List what is available, newest last:

ssh -i /root/.ssh/mcp_heezy -o StrictHostKeyChecking=no mcp-admin@192.168.1.21 \
  "ls -la /nfs/heezy/backups/k8s/openbao/"

Choose the newest snapshot taken before the data was lost. If you are unsure when the loss happened, check when the pod was recreated:

kubectl get pod -n openbao openbao-0 -o jsonpath='{.status.startTime}{"\n"}'

Copy it to wherever you are running kubectl:

mkdir -p /tmp/obrestore
scp -i /root/.ssh/mcp_heezy -o StrictHostKeyChecking=no \
  mcp-admin@192.168.1.21:/nfs/heezy/backups/k8s/openbao/openbao-20260816_125018.snap \
  /tmp/obrestore/

Verify it is a real snapshot before you trust it. A valid one is a gzipped tar with exactly these members:

tar tzf /tmp/obrestore/openbao-20260816_125018.snap

meta.json
state.bin
SHA256SUMS
SHA256SUMS.sealed

If that command errors, or the file is only a few hundred bytes, the backup is bad. Try the next one down the list.

Step 5: Copy the snapshot into the pod¶

sha256sum /tmp/obrestore/openbao-20260816_125018.snap

kubectl cp /tmp/obrestore/openbao-20260816_125018.snap \
  openbao/openbao-0:/tmp/openbao.snap

kubectl exec -n openbao openbao-0 -- sha256sum /tmp/openbao.snap

The two checksums must match. kubectl cp streams through a tar pipe and can truncate silently on a flaky connection. If they differ, delete the file in the pod and copy again.

Step 6: Initialize with temporary keys, then restore¶

This is the part that reads wrong the first time. To load a snapshot, OpenBao must be initialized and unsealed, because snapshot restore is an authenticated API call. So you initialize the empty instance to get a throwaway root token, use it to load the snapshot, and the snapshot then replaces that barrier with the original one.

6a. Initialize. Key shares and threshold must match the original, 3 and 2:

kubectl exec -n openbao openbao-0 -- sh -c \
  "BAO_ADDR=http://127.0.0.1:8200 bao operator init -key-shares=3 -key-threshold=2 -format=json > /tmp/init.json"

kubectl exec -n openbao openbao-0 -- cat /tmp/init.json

Copy the three unseal_keys_b64 values and the root_token into a scratch buffer. They live for the next three commands only. Do not put them in openbao-init-keys.

6b. Unseal with two of the temporary keys. Unsealing takes two separate calls:

kubectl exec -n openbao openbao-0 -- sh -c \
  "export BAO_ADDR=http://127.0.0.1:8200; \
   bao operator unseal '<TEMP_KEY_1>' >/dev/null; \
   bao operator unseal '<TEMP_KEY_2>'"

Confirm Sealed false in the output before continuing.

6c. Restore the snapshot. -force is required because the snapshot's cluster ID does not match this freshly initialized one:

kubectl exec -n openbao openbao-0 -- sh -c \
  "export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='<TEMP_ROOT_TOKEN>'; \
   bao operator raft snapshot restore -force /tmp/openbao.snap"

Success is silent. Exit code 0 and no output is what you want.

The next command will fail, and that is correct

Running bao status now returns 500 ... decryption failed: cipher: message authentication failed. That is the expected result. The restored barrier uses the original encryption keys, and the process is still holding the temporary ones in memory. Step 7 fixes it.

Step 7: Restart and unseal with the original keys¶

kubectl delete pod -n openbao openbao-0

Wait for it to come back. The Longhorn volume has to detach from the old node and attach to the new one, so allow up to 2 minutes:

kubectl get pods -n openbao -l app.kubernetes.io/name=openbao -o wide -w

Once it is Running, confirm the restore took:

kubectl exec -n openbao openbao-0 -- sh -c "BAO_ADDR=http://127.0.0.1:8200 bao status"

You should now see Initialized true and Sealed true. Unseal with the original keys straight from the secret:

K1=$(kubectl get secret openbao-init-keys -n openbao -o jsonpath='{.data.unseal-key-1}' | base64 -d)
K2=$(kubectl get secret openbao-init-keys -n openbao -o jsonpath='{.data.unseal-key-2}' | base64 -d)

kubectl exec -n openbao openbao-0 -- sh -c \
  "export BAO_ADDR=http://127.0.0.1:8200; \
   bao operator unseal '$K1' >/dev/null; \
   bao operator unseal '$K2'"

Sealed false means the restore worked and the original barrier is back. If these keys are rejected, the snapshot you restored predates the current openbao-init-keys secret. Stop and get help rather than re-initializing.

Step 8: Fix the token reviewer JWT¶

Do this every time, even if everything looks healthy.

The Kubernetes auth method stores a service account token that OpenBao uses to call the Kubernetes TokenReview API. In a default install that token is bound to the openbao pod, so deleting the pod in Step 7 invalidates it. Every ExternalSecret then fails with:

Code: 403. Errors:
* permission denied

This is the failure that is easiest to misread, because the login request reaches OpenBao fine. Only the review behind it fails.

Create a service account token secret that never expires and is not tied to a pod:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: openbao-token-reviewer
  namespace: openbao
  annotations:
    kubernetes.io/service-account.name: openbao
type: kubernetes.io/service-account-token
EOF

Wait a few seconds for the controller to populate it, then write it into the auth config:

RT=$(kubectl get secret openbao-init-keys -n openbao -o jsonpath='{.data.root-token}' | base64 -d)
RJWT=$(kubectl get secret -n openbao openbao-token-reviewer -o jsonpath='{.data.token}' | base64 -d | tr -d '\n')

kubectl exec -n openbao openbao-0 -- sh -c \
  "export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='$RT'; \
   bao write auth/kubernetes/config \
     kubernetes_host='https://10.152.183.1:443' \
     token_reviewer_jwt='$RJWT' \
     disable_iss_validation=true \
     kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

10.152.183.1:443 is the in-cluster Kubernetes API service address and does not change.

The openbao service account already has the system:auth-delegator ClusterRole through openbao-server-binding. If that binding is missing, TokenReview will keep returning 403 no matter how fresh the token is.

Step 9: Re-enable the unseal CronJob¶

kubectl patch cronjob -n openbao openbao-unseal -p '{"spec":{"suspend":false}}'

Step 10: Verify¶

Work down this list. Do not stop at the first green check.

10a. OpenBao is unsealed and reachable on its VIP:

curl -sk http://192.168.1.32:8200/v1/sys/health \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('sealed:', d['sealed'])"

10b. The original root token works and the secrets are back:

RT=$(kubectl get secret openbao-init-keys -n openbao -o jsonpath='{.data.root-token}' | base64 -d)

kubectl exec -n openbao openbao-0 -- sh -c \
  "export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='$RT'; \
   bao secrets list; bao kv list secret/production/heezy"

10c. Auth methods, AppRoles and policies survived:

kubectl exec -n openbao openbao-0 -- sh -c \
  "export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='$RT'; \
   bao auth list; bao list auth/approle/role; bao policy list"

Expect four auth methods (approle, kubernetes, oidc, token), two AppRoles (github-heezy, heezy-backup), and six policies (ansible-readonly, default, eso-readonly, heezy-backup, openclaw, root).

10d. Kubernetes auth actually logs in. This is the check that catches a stale reviewer JWT:

JWT=$(kubectl create token external-secrets-sa -n heezy --duration=600s | tr -d '\n')

kubectl run bao-login-test --rm -i --restart=Never -n heezy \
  --image=curlimages/curl:8.10.1 --command -- \
  curl -sk -X PUT http://openbao.openbao.svc.cluster.local:8200/v1/auth/kubernetes/login \
  -H 'Content-Type: application/json' -d "{\"role\":\"eso\",\"jwt\":\"$JWT\"}"

A JSON body containing client_token is success. {"errors":["permission denied"]} means Step 8 did not take.

10e. Every ExternalSecret recovers. The controller caches its provider client, so it does not notice OpenBao came back until it restarts:

kubectl rollout restart deployment -n external-secrets external-secrets

# give it about a minute, then:
kubectl get secretstores -A -o custom-columns=NS:.metadata.namespace,STATUS:.status.conditions[0].reason --no-headers
kubectl get externalsecrets -A -o custom-columns=READY:.status.conditions[0].status --no-headers | sort | uniq -c

Both SecretStores must read Valid, and all 20 ExternalSecrets must read True.

10f. CI can log in again. Every Gitea workflow starts with an AppRole login, so this is the real end to end test. Dispatch any small deploy and confirm the Configure AWS credentials via OpenBao step passes:

curl -sk -u 'heezy-admin:<password>' -X POST \
  "http://192.168.1.31:3000/api/v1/repos/heezy-admin/heezy-containers/actions/workflows/deploy-receipts.yml/dispatches" \
  -H 'Content-Type: application/json' -d '{"ref":"main"}'

A failure here shows up as json.decoder.JSONDecodeError: Expecting value: line 1 column 1, which is the workflow parsing an empty login response.

Step 11: Clean up¶

kubectl exec -n openbao openbao-0 -- rm -f /tmp/openbao.snap /tmp/init.json
rm -rf /tmp/obrestore

The temporary root token and unseal keys from Step 6 are dead the moment the snapshot is restored. Clear them out of your terminal scrollback anyway.

Only sealed, not a restore¶

If bao status shows Initialized true and Sealed true, nothing is lost. The openbao-unseal CronJob unseals it within 5 minutes on its own. To do it immediately:

kubectl create job -n openbao --from=cronjob/openbao-unseal openbao-unseal-manual
kubectl logs -n openbao job/openbao-unseal-manual

Things that will trip you up¶

What you see	What it means
`decryption failed: cipher: message authentication failed` right after the restore	Correct and expected. Restart the pod, then unseal with the original keys.
`Multi-Attach error for volume` on the new pod	Longhorn has not detached from the old node yet. Wait about a minute.
`permission denied` on every ExternalSecret, but OpenBao is unsealed	Stale token reviewer JWT. Step 8.
SecretStore still `InvalidProviderConfig` after Step 8	The controller caches its client. Restart the external-secrets deployment.
`sh: curl: not found` inside the openbao pod	The image has no curl. Use `bao` inside the pod, or run curl from a throwaway pod as in 10d.
Workflows fail with `JSONDecodeError` at the OpenBao step	OpenBao returned an empty body, so it is sealed or down.

What this does not cover¶

The snapshot only holds what was inside OpenBao. If the underlying Longhorn volume was destroyed, the volume itself is recreated empty by the StatefulSet and this runbook refills it, which is fine. But note that Longhorn has no backup target configured, so for any other PVC there is no equivalent of this procedure. See Backups.