Scheduled Job Failed¶

Runbook for the Grafana alert Scheduled Job Failed (k8s-cronjob-failed, group k8s-job-alerts).

What the alert means¶

A Kubernetes Job in the heezy namespace retried until it hit its backoffLimit and gave up. None of the CronJobs set backoffLimit, so the default of 6 applies: the pod failed six times in a row before the Job was marked Failed. That run of the schedule produced nothing. The next run fires on schedule and is unaffected, so a single alert is usually a transient dependency blip, not a broken job.

The alert fires 5 minutes after the failure is observed and clears once the Job ages past 2 hours. It does not clear because anything was fixed. Treat the resolve notification as "stop looking at the clock", not "resolved".

Reading the alert¶

The notification carries two labels:

Label	Example	Meaning
`cronjob`	`heezy-finance-sync`	The schedule that broke. This is what you act on.
`job_name`	`heezy-finance-sync-29773320`	The single Job object. The suffix is a unix-minute stamp, not a version.

Get the logs¶

Do not use kubectl logs. The alert has a 5 minute for clause and Kubernetes garbage collects the failed pod well before that, so by the time the message reaches Discord the pod is gone and kubectl logs -n heezy job/<name> returns error: timed out waiting for the condition.

The logs are in Loki, which promtail ships from every pod in the namespace. Retention is 3 days.

Grafana → Explore → Loki datasource:

{namespace="heezy", pod=~"heezy-finance-sync-29773320-.+"}

Drop the Job suffix to see every run of a schedule:

{namespace="heezy", pod=~"heezy-finance-sync-.+"} |= "Fatal"

If the failure is older than 3 days, the logs are gone everywhere. Skip to Confirm current health and decide whether it is still happening.

Confirm current health¶

The Job object survives the pod. failedJobsHistoryLimit is 3 on every CronJob and no ttlSecondsAfterFinished is set, so up to three failed Jobs per schedule sit in the namespace indefinitely. They are history, not active problems.

kubectl get jobs -n heezy --sort-by=.metadata.creationTimestamp

Compare the AGE column against the schedule. If the newest Job for that CronJob is Complete, the schedule has recovered on its own and there is nothing to do beyond noting the cause.

The schedules¶

CronJob	Schedule (UTC)	Runs	Depends on
`heezy-finance-sync`	`0 * * * *`	`amazon_orders.py --hours 2`	Gmail API, Ollama on big-boi, Postgres
`heezy-statement-scanner`	`30 * * * *`	`scan_statements.py`	`nfs-heezy-ingest` PVC, Postgres
`heezy-budget-alerts`	`0 14 * * *`	`budget_alerts.py`	Postgres, Discord webhook
`heezy-mailbot-amazon-forward`	`/5 * * *`	`app.jobs.amazon_forward --live`	Gmail API
`heezy-mailbot-receipt-ingest`	`/15 * * *`	`app.jobs.receipt_ingest --live`	Gmail API, receipts app
`heezy-mailbot-triage`	`0 * * * *`	`app.jobs.triage --live --hours 2`	Gmail API
`ecr-credentials-refresh`	`0 /6 * *`	rewrites the `ecr-credentials` secret	AWS ECR, in-cluster RBAC
`plex-config-backup`	`0 4 * * *`	tars Plex config to NFS	`plex-config` PVC, NFS
`uptime-kuma-backup`	`0 3 * * *`	copies `kuma.db` to NFS	`uptime-kuma-data` PVC, NFS

Details: Mailbot, Data Sources, Backups.

Common causes¶

Symptom in the logs	Cause	Fix
`Fatal: missing Gmail env credentials`	The ExternalSecret did not project `GMAIL_CLIENT_ID` / `GMAIL_CLIENT_SECRET` / `GMAIL_REFRESH_TOKEN`	Check the ExternalSecret synced, then the OpenBao path
`invalid_grant`	Gmail refresh token revoked or expired	Rotate the token, see Mailbot
Connection refused to `192.168.1.21:11434`	Ollama down on big-boi	Restart Ollama. `heezy-finance-sync` cannot parse order emails without it
Connection refused to `192.168.1.21:5432`	Postgres down or connection limit reached	Check big-boi
`ImagePullBackOff` on the pod, no application logs	ECR token expired	Check `ecr-credentials-refresh` ran in the last 6h. Its own failure cascades into every other job on the `heezy-finance` and `heezy-mailbot` images
Nothing in Loki at all	Pod never started	`kubectl describe job -n heezy <job_name>` and read the events

An ecr-credentials-refresh failure is the one that fans out. Check it first when several unrelated schedules alert at once.

Rerun a schedule by hand¶

CronJobs are declarative in heezy-k8s, so trigger a one-off Job from the CronJob rather than editing anything:

kubectl create job -n heezy --from=cronjob/heezy-finance-sync manual-$(date +%s)

Delete the manual Job when you are done reading its logs. It does not count against failedJobsHistoryLimit and will otherwise linger.

Clearing stale failed Jobs¶

Old Failed Job objects are cosmetic but they clutter kubectl get jobs and keep kube_job_status_failed at 1. The alert expression already ignores anything older than 2 hours, so there is no need to clean them up to silence anything.

Deleting them is destructive and irreversible. State the exact Jobs first, then:

kubectl delete job -n heezy <job_name>

How the alert is built¶

The rule lives in ansible-heezy/roles/lgtm/templates/grafana-alerting.yml.j2 and is deployed by the lgtm playbook. See Monitoring.

max by (cronjob, job_name) (
  (kube_job_status_failed{namespace="heezy"} > 0)
  * on (job_name) group_left (cronjob)
    label_replace(kube_job_owner{namespace="heezy", owner_kind="CronJob"},
                  "cronjob", "$1", "owner_name", "(.*)")
  * on (job_name) group_left ()
    (kube_job_status_start_time{namespace="heezy"} > bool time() - 7200)
)

Three things are load bearing:

kube_job_owner join. kube_job_status_failed only carries job_name, which is stamped with a unix minute and does not identify the schedule. The join adds cronjob.
> bool, not and. The 2 hour window exists so a stale failed Job does not alert forever. It used to be written as and on(job_name) (kube_job_status_start_time > time() - 7200), which removed the series once the job aged out. Grafana then resolved the instance as MissingSeries or NoData and re-rendered the annotations against an empty label set, so every resolve notification read CronJob [no value] in namespace heezy exhausted its backoff limit. Multiplying by a bool keeps the series present at value 0, and the resolve carries its labels.
> 0 before the joins. Without it every Job in the namespace produces an instance, including the successful ones, and Grafana accumulates dozens of Normal instances.

[no value] in an old notification

Discord messages sent before 2026-08-14 show CronJob [no value]. That is the pre-fix resolve behaviour described above, not a live problem. The job_name was never in that message, so those alerts cannot be traced back to a schedule after the fact.