Skip to content

Gmail Token Rotation

Runbook for the Grafana alert Gmail OAuth Token Expired (gmail-oauth-token-expired, group oauth-alerts).

The short version

Go to oauth.heezy.info/rotate, click Authorize with Google, sign in as the heezy Gmail account, accept both scopes. Done. Everything below is why, and what to do when that does not work.

What the alert means

Google's token endpoint returned invalid_grant to something in the heezy namespace. The refresh token stored at OpenBao production/heezy/gmail/oauth is dead, and every Gmail-backed workload is down at once:

Workload Schedule What stops
heezy-mailbot-amazon-forward every 5 min Order confirmations stop being forwarded
heezy-mailbot-receipt-ingest every 15 min Receipts stop reaching receipts.heezy.info
heezy-mailbot-triage hourly Inbox stops being triaged
heezy-finance-sync hourly Orders stop loading into heezy-finance

One secret, one key, four consumers. There is nothing to fix per-app.

Why this alert exists

Before it, the same failure paged as Container Restart Spike and Scheduled Job Failed — two symptoms with a dozen possible causes, neither of which mentions a token. A silent version of this outage ran from 2026-08-03 to 2026-08-10 before anyone noticed.

Why it keeps happening

The OAuth client lives in GCP project onyx-incentive-497913-i9 (still named "OpenClaw"). Its consent screen is in Testing mode, and Google expires refresh tokens issued by a Testing-mode client after 7 days. That is the entire cause. See Make it stop for the permanent fix.

Rotate the token

1. Open the rotation page

https://oauth.heezy.info/rotate

Nothing to unlock. /rotate is behind Cloudflare Access on the path, using the same Google login and the same home-IP bypass as finance, receipts and the rest, so from the house it just opens.

The host itself cannot go behind Access: /callback is where Google's redirect lands, and an identity challenge there breaks every flow the service exists to run. Only /rotate is protected.

Access is not enforced on the LAN

oauth.heezy.info resolves to the SWAG VIP internally, so LAN traffic never reaches the Cloudflare edge. That is deliberate and safe here: the account check is the real boundary. A rotation only lands if the token is a live Google grant for the allow-listed account under our own client ID, which no network position lets you forge. Access is what stops the page being readable from the public internet.

2. Read the status card

The page probes the stored token against Google on every load, so the badge is live, not cached:

Badge Meaning
working The stored token still refreshes. Nothing to do.
expired invalid_grant. Rotate now.
unknown The page could not reach OpenBao or Google. Fix that first, see Troubleshooting.

It also shows the token's age in days. Anything at 6+ days is about to expire.

3. Authorize with Google

Click Authorize with Google. The link already carries access_type=offline and prompt=consent, which is what makes Google mint a new refresh token instead of handing back the dead one.

Accept both scopes

gmail.modify and gmail.send. gmail.send is not implied by gmail.modify. A token missing it passes every check the mailbot makes at startup and then fails at the first forward.

The page rejects a token missing either scope before it is stored, so you cannot get this wrong through /rotate — but you can through a hand-rolled flow.

Sign in as the heezy Gmail account. A token for any other account is rejected: /authorize is public, so without that check anyone completing the consent flow would overwrite production.

4. Nothing to paste

The callback writes the new refresh token straight into OpenBao, merging it into the existing secret so client_id, client_secret and forward_to survive. You land back on /rotate with a green banner and the badge flipped to working.

5. Wait 5 minutes, or don't

The two Gmail ExternalSecrets run at refreshInterval: 5m specifically so a rotation lands before the next scheduled run. Note this updates the Secret, not running pods: env from secretKeyRef is injected at pod creation, so a pod that started before the rotation keeps replaying the dead token through its restarts until its Job gives up. The next scheduled run picks up the new one.

To skip the wait:

kubectl annotate externalsecret -n heezy heezy-mailbot-gmail heezy-finance-gmail force-sync=$(date +%s) --overwrite

Confirm the propagation:

kubectl get secret heezy-mailbot-gmail -n heezy -o jsonpath='{.data.GMAIL_REFRESH_TOKEN}' | base64 -d | head -c 12; echo

6. Verify

amazon-forward runs every 5 minutes, so it is the fastest proof. Once it is green, decide whether anything was lost: Recover the window you lost.

kubectl logs -n heezy -l job-name=$(kubectl get jobs -n heezy -o name | grep amazon-forward | tail -1 | cut -d/ -f2) --tail=20

"success": true and no error key. A CrashLoopBackOff pod from before the rotation is not cleaned up automatically — delete it rather than waiting:

kubectl delete pod -n heezy --field-selector status.phase=Failed

The paste box

The second card on /rotate takes a refresh token you obtained elsewhere — the Google OAuth Playground, a different browser, a flow you ran by hand. Use it when the redirect flow cannot complete, for example when oauth.heezy.info is unreachable from where you are but you can still reach it over the LAN.

The token is checked against Google before anything is written: it must refresh successfully, carry both required scopes, and belong to the allow-listed account. A truncated copy/paste is rejected on the spot instead of by an alert an hour later.

The write is a compare-and-swap against the OpenBao version that was read, so a concurrent rotation fails loudly rather than silently winning.

Troubleshooting

Badge says unknown, probe error mentions OpenBao

The service authenticates to OpenBao with a dedicated AppRole (oauth-rotator) scoped to write only secret/data/production/heezy/gmail/oauth. It logs in on every call, so a rotated secret_id breaks rotation immediately and nothing else.

kubectl logs -n heezy deploy/oauth-callback --tail=50
kubectl get secret oauth-callback-secrets -n heezy -o jsonpath='{.data.VAULT_ROLE_ID}' | base64 -d; echo

The page asks for a Google login and will not take yours

That is Cloudflare Access, not the app. The allow policy uses local.allowed_emails in terraform-heezy/environments/production/cloudflare/access.tf, shared with every other heezy app. From the house the bypass policy should mean no prompt at all — if you are seeing one there, check whether local.home_ip still matches your WAN address.

Re-issue the secret_id with the root token (from openbao-init-keys, see the OpenBao runbook), write it back to production/heezy/oauth-callback/config, then restart the deployment.

"That token belongs to X, not Y"

You signed in with the wrong Google account. The expected account is the ALLOWED_GOOGLE_ACCOUNT key in production/heezy/oauth-callback/config. Note this is the mailbox being read, not forward_to, which is the address Amazon confirmations are forwarded to.

"Could not obtain refresh token"

Google only returns a refresh token when prompt=consent is present. If you built the authorize URL yourself, you left it off. Use the button on /rotate.

Rotation worked but jobs still fail

Check that the ExternalSecret actually synced and that you are not looking at a pre-rotation pod:

kubectl get externalsecret -n heezy heezy-mailbot-gmail -o jsonpath='{.status.conditions[0]}'; echo
kubectl get pods -n heezy | grep -E "mailbot|finance"

If heezy-finance-sync still fails while mailbot recovers, it is a different bug — the sync exits 0 on an empty result set, so a failing sync is not a token problem. See Scheduled Job Failed.

Recover the window you lost

Restoring the token restores service. It does not go back for the mail that arrived while the token was dead. Whether anything was actually lost depends entirely on which job, because the lookback windows differ by two orders of magnitude.

Job Looks back Runs every Survives an outage of
amazon_forward 3 days (--max-age-days) 5 min 3 days
receipt_ingest 7 days (hardcoded newer_than:7d) 15 min 7 days
triage 2 hours (--hours 2) hourly 2 hours
heezy-finance-sync 2 hours (--hours 2) hourly 2 hours

First, measure the outage

Loki keeps 3 days. The first and last invalid_grant bracket it exactly:

{namespace="heezy"} |= "invalid_grant"

If the span is under 2 hours, stop — every job self-heals on its next run and there is nothing to do.

The day count is not what protects you

amazon_forward and receipt_ingest both gate on is:unread, not on the ledger. Reading an order confirmation in the Gmail app takes it out of the query permanently, whether it is three hours or three days old. So the 3 and 7 day windows only hold if nobody touched the mail in the meantime, which during a multi-hour outage is exactly the sort of thing that happens.

Check what the query would actually match before assuming those two jobs caught up. seen: 0 in the run output is not evidence either way — it means the query matched nothing, which is equally consistent with "nothing to do" and "the mail dropped out of the window".

heezy-finance-sync is the one that loses data

The code default is --hours 168. The CronJob overrides it to 2. Nothing re-scans, so any outage longer than 2 hours is a permanent hole in orders unless you backfill it.

Widening the CronJob is the wrong fix: each run would re-parse a day of mail through Ollama at 15 to 20 seconds a message, every hour, forever, to cover an event that happens weekly at worst. Backfill on demand instead.

--from=cronjob/... cannot override the command (error: cannot specify --from and command), so copy the job template and swap the args:

kubectl get cronjob -n heezy heezy-finance-sync -o json | python3 -c "
import json,sys,time
c = json.load(sys.stdin)
j = c['spec']['jobTemplate']
j['apiVersion'], j['kind'] = 'batch/v1', 'Job'
j['metadata'] = {'name': 'finance-backfill-%d' % time.time(), 'namespace': 'heezy'}
j['spec']['template']['spec']['containers'][0]['command'] = [
    'python3', 'amazon_orders.py', '--hours', '24', '--dry-run']
j['spec']['ttlSecondsAfterFinished'] = 3600
json.dump(j, sys.stdout)" | kubectl create -f -

Set --hours to comfortably more than the outage. Re-scanning is safe: order_id is the primary key and _normalize_order_id() runs before dedupe, so already-ingested orders are skipped.

Drop --dry-run only after reading the output

A real run archives Amazon mail out of the inbox, and that is the one step you cannot undo. The dry run inserts into the open transaction so in-run dedupe still behaves, rolls back, and skips every Gmail mutation. Read what it says it would add, then re-run without the flag.

kubectl logs -n heezy job/finance-backfill-<stamp> --tail=40

triage is not worth backfilling

Its 2 hour window means a long outage leaves mail untriaged, but the only consequence is that it stays in the inbox. Archiving it later by hand is fine. Do not widen --hours to chase it — triage trashes as well as archives, and a wide window replays those decisions over mail you have since dealt with yourself.

Make it stop permanently

Publishing the consent screen removes the 7 day expiry entirely and retires both this runbook and its alert.

  1. GCP console → project onyx-incentive-497913-i9 → APIs & Services → OAuth consent screen.
  2. Publishing status Testing → In production.
  3. The app uses restricted Gmail scopes, so Google shows an unverified-app warning to any account outside the test users list. For a single-owner internal app that is cosmetic: click through "Advanced → Go to (unsafe)" once at consent time. Full verification is only needed to remove the warning for other people.

Until that happens the rotation is weekly, and the alert is what tells you it is due.

The other button on that page

/rotate also carries Authenticate with Spotify, which has nothing to do with Gmail. It re-authenticates Lidarr's Spotify Playlists import list. Nothing in this runbook applies to it: no OpenBao path, no External Secrets, no Gmail job affected. What breaks when it lapses is the playlist import, and the symptom in Lidarr's log is

[Warn] HttpClient: HTTP Error - Res: HTTP/2.0 [GET] https://spotify.lidarr.audio/renew?refresh_token=(removed) 400.BadRequest
{"error":"invalid_grant","error_description":"Refresh token expired"}
[Error] FetchAndParseImportListService: Error during Import List Sync of Spotify Playlists

invalid_grant is the RFC 6749 error code, not a Google string. Every OAuth provider emits it verbatim, which is why the gmail-oauth-token-expired alert is scoped to the pods that read the Gmail secret rather than to the namespace.

The button cannot mint a Spotify token of its own. Lidarr renews at spotify.lidarr.audio/renew, a broker holding Lidarr's client secret, so a grant issued under any other client ID would last one hour and then fail permanently. /spotify/authorize therefore drives Lidarr's own startOAuth and getOAuthToken actions and writes the result back through Lidarr's API, exactly as its settings UI does. The click is the same either way; this one just lives next to the Gmail button.

The status card reads Lidarr's stored config and nothing more. The expires it shows is the one hour access token, so a timestamp in the past there is normal and not a reason to press the button. Press it when a sync fails with invalid_grant.

Lidarr validates against Spotify before saving, so a grant that does not work is rejected at the click with Spotify authentication error, not stored and left to fail at the next sync.