Adding a Scheduled Job

The shape every recurring job in the lab uses: a Python app in heezy-containers, an image in ECR, and a CronJob in heezy-k8s. No Deployment, no Service, no SWAG conf, no DNS entry.

Use this instead of New Service when the thing runs on a schedule and then exits. Use New Service when it listens on a port.

Jobs that follow this pattern

CronJob	Source	Schedule	What it does
heezy-mailbot-amazon-forward	heezy-mailbot	*/5 * * * *	Forwards Amazon order mail
heezy-mailbot-receipt-ingest	heezy-mailbot	*/15 * * * *	Pushes receipts to receipts.heezy.info
heezy-mailbot-triage	heezy-mailbot	0 * * * *	Archives and trashes inbox mail
heezy-finance-sync	heezy-finance	0 * * * *	Refreshes finance source data
heezy-budget-alerts	heezy-finance	0 14 * * *	Posts budget alerts to Discord
heezy-statement-scanner	statements	30 * * * *	Picks up new statement uploads
music-sweeper	music-sweeper	*/30 * * * *	Sweeps slskd downloads into Plex
plex-config-backup	in-manifest	0 4 * * *	Backs up the Plex config PVC
uptime-kuma-backup	in-manifest	0 3 * * *	Backs up Uptime Kuma
ecr-credentials-refresh	in-manifest	0 */6 * * *	Refreshes the ECR pull secret

The first seven build from heezy-containers. The last three are small enough to live entirely in their manifest and run a stock image with a shell command.

Layout

heezy-containers/dockerfiles/<app>/
  Dockerfile
  requirements.txt
  README.md
  app/
    __init__.py
    config.py          # env parsing, one class, no imports from the rest of app/
    <domain modules>
    <entrypoint>.py    # main(argv=None) -> exit code
  tests/
    conftest.py
    test_*.py

heezy-containers/.gitea/workflows/
  deploy-<app>.yml     # build + push + trigger, on push to main
  test-<app>.yml       # pytest + coverage gate, on PR and non-main pushes

heezy-k8s/apps/<app>/
  cronjob.yaml
  external-secret.yaml
  kustomization.yaml

1. ECR repository

The repository name must start with heezy-. This is not a style preference: the runner's github-runner-ecr-push IAM policy scopes every ECR write action to arn:aws:ecr:us-east-2:025066240222:repository/heezy-*. An unprefixed name fails twice — terraform apply cannot create it (AccessDeniedException on ecr:CreateRepository) and the build cannot push to it (denied on ecr:InitiateLayerUpload) — and neither error mentions the wildcard, so it reads like a missing action rather than a missing prefix.

The dockerfiles/ directory does not need the prefix. receipts and statements both keep a short directory alongside a heezy--prefixed image.

terraform-heezy/environments/production/aws/ecr.tf:

resource "aws_ecr_repository" "my_job" {
  name                 = "heezy-my-job"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "aws_ecr_lifecycle_policy" "my_job" {
  repository = aws_ecr_repository.my_job.name

  policy = jsonencode({
    rules = [{
      rulePriority = 1
      description  = "Keep last 10 images"
      selection = {
        tagStatus   = "any"
        countType   = "imageCountMoreThan"
        countNumber = 10
      }
      action = { type = "expire" }
    }]
  })
}

This must merge and apply before the build workflow runs, or the push fails on a missing repository. Merging the two PRs seconds apart is not enough — the terraform apply takes a couple of minutes, and the build starts immediately. Confirm the repository exists before merging the app PR:

aws ecr describe-repositories --region us-east-2 --repository-names heezy-my-job

If the build already failed this way, fixing it needs a dispatch rather than a retry: deploy-<app>.yml filters on dockerfiles/<app>/**, so a commit that only corrects the image name in the workflow triggers nothing. Give every deploy workflow a workflow_dispatch: trigger so this is recoverable without an empty commit.

2. The app

Dockerfile is the same eleven lines every time. The only decision is the default CMD:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ /app/app/
ENV PYTHONUNBUFFERED=1
CMD ["python3", "-m", "app.myjob", "--dry-run"]

Make the safe mode the image default. Every CronJob overrides command anyway, so the CMD only ever runs when someone pulls the image and runs it by hand. heezy-mailbot requires an explicit --dry-run or --live and defaults to neither; music-sweeper defaults to --dry-run. Both are fine. Defaulting to the destructive path is not.

config.py

One class, everything from the environment, defaults that match production so a bare run in the container does the right thing. Take the environment as an argument so tests can construct one without touching os.environ:

class Config:
    def __init__(self, env=None):
        env = env if env is not None else os.environ
        self.batch_size = _int(env, "BATCH_SIZE", 25)
        self.dry_run = _bool(env, "DRY_RUN", False)

A helper that reads os.environ directly instead of env is the easiest bug to write here and it only shows up in tests.

Path variables come in pairs

When the job and another service share a PVC at different mount paths, both strings have to be configured. The job cannot derive the other service's view.

- name: DOWNLOADS_DIR          # this pod's mount
  value: "/downloads"
- name: LIDARR_DOWNLOADS_DIR   # the same PVC as Lidarr mounts it
  value: "/slskd-downloads"

soularr documents this trap in its own ConfigMap after getting it wrong. Change one mount, change both values.

3. Workflows

Copy deploy-heezy-mailbot.yml and test-heezy-mailbot.yml, then search-and-replace the app name. Both are path-filtered to dockerfiles/<app>/**, so they stay quiet for unrelated commits.

Workflows go in .gitea/workflows/, never .github/workflows/.

The deploy workflow ends by committing a .deploy-trigger file into heezy-k8s/apps/<app>/, which is what actually causes the cluster to pick up the new image. auto-deploy.yaml in heezy-k8s applies every apps/*/ that has a kustomization.yaml, so no change there is needed for a new job.

Coverage gate

test-<app>.yml runs pytest --cov=app --cov-fail-under=<N>. Pick the floor from what the app actually reaches on the first green run, rounded down. Current floors: heezy-finance 13, receipts 88, heezy-projects 62, heezy-maintenance 90, heezy-mailbot 90, music-sweeper 90.

An app with no tests/ directory gets no status check at all, which means branch protection will happily merge a broken image. Write the tests.

4. Secrets

Never a literal in the manifest, never a committed Secret. An ExternalSecret pulls from OpenBao:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-job-secrets
  namespace: heezy
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: openbao
    kind: SecretStore
  target:
    name: my-job-secrets
    creationPolicy: Owner
  data:
    - secretKey: API_KEY
      remoteRef:
        key: all/heezy/media/lidarr
        property: api_key

Then reference it with secretKeyRef. Mark it optional: true when the job should still run without it, as with a Discord webhook.

See Secrets & Access for writing the value into OpenBao first. An ExternalSecret pointing at a path that does not exist leaves the k8s Secret uncreated and the pod stuck in CreateContainerConfigError.

5. The CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: my-job
  namespace: heezy
spec:
  schedule: "*/30 * * * *"
  concurrencyPolicy: Forbid        # non-negotiable for anything that writes
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          imagePullSecrets:
            - name: ecr-credentials
          restartPolicy: OnFailure
          containers:
            - name: my-job
              image: 025066240222.dkr.ecr.us-east-2.amazonaws.com/my-job:latest
              imagePullPolicy: Always
              command: ["python3", "-m", "app.myjob", "--live"]

imagePullSecrets: ecr-credentials and imagePullPolicy: Always are both required. The pull secret is rotated every six hours by ecr-credentials-refresh; without it the pod fails to pull once the current token expires rather than immediately, which makes it look intermittent.

6. Notification

A job nobody watches needs to report itself. The convention is a Discord embed at the end of the run, pulled from all/heezy/discord/webhooks in OpenBao.

Existing webhook keys: ansible-heezy, claude-code, heezy-containers, heezy-k8s, terraform-heezy, mittentech-production-infrastructure. Reuse the one matching the repo the job deploys from, or add a key for a dedicated channel.

Rules that keep this useful:

• Never raise from the notifier. A missed ping must not fail a run that otherwise succeeded. Catch the transport error and return False.
• Put the whole story in the embed. Counts per outcome, what was changed, what was skipped and why. If the reader has to open kubectl logs, the embed failed.
• Colour by severity so a glance is enough: green for a clean run, orange when something was deleted, red on errors.
• Truncate lists. Discord caps an embed field at 1024 characters. Show the first handful and a +N more line.

Writing a job that is safe to run unattended

The jobs that have caused problems in this lab all failed the same way: they ran against a state they had not verified, or they ran twice.

• Idempotency belongs in a table, not in a flag. heezy-mailbot writes mailbot_processed (job, message_id) before the side effect, not after. The n8n workflows it replaced used is:unread plus a mark-as-read call, so any crash between the two repeated the side effect forever.
• Verify external state before acting on it. music-sweeper asks slskd which files are in flight and stops the entire run if slskd cannot be reached, because "no answer" and "nothing active" are not the same thing.
• Cap the batch. A per-run limit bounds both blast radius and load on whatever the job talks to. It also makes the first live run cheap to inspect.
• Fail closed on ambiguity. When a record cannot be identified confidently, skip and report it. Never take the destructive branch on a guess.
• Take a lock if a human might run it too. concurrencyPolicy: Forbid covers scheduled overlap only. A lock file covers kubectl create job --from=cronjob/... landing mid-run.

Verifying

kubectl get cronjob -n heezy my-job
kubectl create job -n heezy --from=cronjob/my-job my-job-manual-$(date +%s)
kubectl logs -n heezy -l job-name=my-job-manual-<ts> --tail=50

Never kubectl apply the manifests by hand. Push to Gitea and let auto-deploy.yaml do it.

When a scheduled run goes wrong, see Scheduled Job Failed.