CI/CD Runners¶

Gitea act-runners running in Kubernetes handle all CI/CD. GitHub Actions are not used — GitHub is a read-only mirror.

Last verified: 2026-08-22.

Quick Reference¶

Setting	Value
Gitea URL (internal)	`http://gitea.heezy.local:3000` (MetalLB VIP)
Gitea URL (from a job)	`http://gitea.heezy.local:3000` — same name; the NodePort is no longer needed
`DOCKER_HOST` (build steps)	`tcp://172.17.0.1:2375`
`DOCKER_HOST` (runner container)	`tcp://localhost:2375`
Runner labels	`ubuntu-latest`, `ubuntu-22.04`, `self-hosted` — all the same image
Job container image	`025066240222.dkr.ecr.us-east-2.amazonaws.com/heezy-runner:latest`
Concurrency	3 pods × `capacity: 2` = 6 concurrent jobs
Job timeout	3h (runner-level), overridden per workflow with `timeout-minutes`
Runner namespace	`gitea`

Never use gitea.heezy.info:3000 from a job. That name resolves to SWAG, which does not proxy git.

Architecture¶

  Gitea (http://gitea.heezy.local:3000  ->  192.168.1.31)
       |
       | act-runner registration
       |
  ┌────▼─────────────────────────────────────────────┐
  │  K8s Deployment: act-runner (gitea namespace)    │
  │  3 replicas, one per node                        │
  │                                                  │
  │  Pod (×3):                                       │
  │  ┌──────────────────┐  ┌──────────────────────┐  │
  │  │  act_runner:latest│  │  docker:27-dind       │  │
  │  │  capacity: 2      │◄─┤  (privileged)         │  │
  │  │                   │  │  DOCKER_HOST exposed  │  │
  │  │                   │  │  at tcp://172.17.0.1  │  │
  │  └──────────────────┘  └──────────────────────┘  │
  └──────────────────────────────────────────────────┘
       |
       | job container pulled from ECR
       |
  ┌────▼──────────────────────────────────────────────┐
  │  heezy-runner:latest (ECR, us-east-2)             │
  │  ubuntu:22.04 + terraform 1.12.2, kubectl v1.31.0,│
  │  aws CLI v2, ansible-core + collections,          │
  │  docker CLI, pg_dump 16, git, python3, jq, sshpass│
  └───────────────────────────────────────────────────┘

heezy-runner Image¶

All three runner labels resolve to the same custom ECR image:

Image: 025066240222.dkr.ecr.us-east-2.amazonaws.com/heezy-runner:latest
Built by: build-heezy-runner.yml in ansible-heezy, from Dockerfile.runner
Pinned versions: terraform 1.12.2, kubectl v1.31.0, pg_dump from the PostgreSQL 16 apt repo so dumps match the server
No app code baked in — repos are always cloned at runtime from Gitea

This replaced the old ansible-automation image + catthehacker/ubuntu:act-22.04 approach.

kubectl is one minor behind the cluster

The image ships kubectl v1.31.0 against a v1.32.9 cluster. That is inside the supported ±1 skew, but it means a v1.32-only flag will fail in CI while working from your laptop.

ECR Pull Auth¶

Runner pods need ECR credentials to pull the job image:

ecr-refresh-aws-creds secret — AWS creds for ECR login (created manually once)
ecr-credentials secret — docker-registry secret, refreshed every 6h by the ecr-credentials-refresh CronJob (one in gitea, one in heezy)
imagePullSecrets in the act-runner deployment references ecr-credentials

An expired ecr-credentials shows up as ImagePullBackOff on job pods, not as a workflow error.

Workflow Authoring Rules¶

Push to Gitea, not GitHub¶

git push gitea main && git push origin main

In heezy-docs the Gitea remote is called origin, so there git push origin main is the correct, pipeline-triggering push. See CI/CD.

VAULT_TOKEN must not cross step boundaries¶

The act-runner redacts masked values written to $GITHUB_ENV. Never do:

# BROKEN — next step reads VAULT_TOKEN=***
echo "VAULT_TOKEN=$VAULT_TOKEN" >> $GITHUB_ENV

Fetch and use the token in the same shell step.

DOCKER_HOST required for docker commands¶

Job steps run directly in heezy-runner. Steps that call docker build/docker push must set:

env:
  DOCKER_HOST: tcp://172.17.0.1:2375

Ansible and terraform steps do NOT need this.

Workflow files location¶

All workflows go in .gitea/workflows/ only. Never use .github/workflows/ — the act-runner would execute both, causing duplicate runs. ansible-heezy does have a .github/ directory, but it holds scripts/ only.

Shell compatibility¶

Act-runner uses /bin/sh, not bash. Replace all [[ ]] constructs:

# BROKEN in sh
EMOJI=$([[ "$STATUS" == "success" ]] && echo "ok" || echo "fail")

# Works in sh
if [ "$STATUS" = "success" ]; then EMOJI="ok"; else EMOJI="fail"; fi

Git clone target¶

git clone http://heezy-admin:${{ secrets.GIT_CLONE_TOKEN }}@gitea.heezy.local:3000/heezy-admin/<repo>.git

heezy-containers, heezy-k8s and heezy-docs workflows still clone from 192.168.1.31:3000. Both work; the name is the form to use in anything new.

Ansible Workflow Pattern¶

Generated by scripts/generate-workflows.py — do not hand-write these. Shown here so you can read one:

- name: Run playbook
  env:
    OPENBAO_ADDR: http://openbao.heezy.local:8200
    OPENBAO_ROLE_ID: ${{ secrets.OPENBAO_ROLE_ID }}
    OPENBAO_SECRET_ID: ${{ secrets.OPENBAO_SECRET_ID }}
    ANSIBLE_HOST_KEY_CHECKING: "False"
    GITEA_SHA: ${{ github.sha }}
  run: |
    VAULT_TOKEN=$(curl -sk -X POST "$OPENBAO_ADDR/v1/auth/approle/login" \
      -H 'Content-Type: application/json' \
      -d "{\"role_id\":\"$OPENBAO_ROLE_ID\",\"secret_id\":\"$OPENBAO_SECRET_ID\"}" \
      | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

    UBUNTU_CREDS=$(curl -sk "$OPENBAO_ADDR/v1/secret/data/production/heezy/ubuntu/cloud-init-credentials" \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      | python3 -c "import sys,json; d=json.load(sys.stdin)['data']['data']; print(d['username'],d['password'])")
    export ANSIBLE_REMOTE_USER=$(echo $UBUNTU_CREDS | cut -d' ' -f1)
    export ANSIBLE_REMOTE_PASSWORD=$(echo $UBUNTU_CREDS | cut -d' ' -f2-)

    git clone http://heezy-admin:${{ secrets.GIT_CLONE_TOKEN }}@gitea.heezy.local:3000/heezy-admin/ansible-heezy.git /tmp/ansible-heezy
    cd /tmp/ansible-heezy && git -c advice.detachedHead=false checkout $GITEA_SHA

    VAULT_TOKEN=$VAULT_TOKEN ansible-playbook \
      -i /tmp/ansible-heezy/inventory/hosts.yml \
      /tmp/ansible-heezy/playbooks/<name>.yml

Terraform Workflow Pattern¶

One workflow per workspace, named terraform-<env>-<provider>-execution.yml:

- name: Terraform apply
  env:
    OPENBAO_ADDR: http://openbao.heezy.local:8200
    OPENBAO_ROLE_ID: ${{ secrets.OPENBAO_ROLE_ID }}
    OPENBAO_SECRET_ID: ${{ secrets.OPENBAO_SECRET_ID }}
  run: |
    VAULT_TOKEN=$(curl -sk -X POST "$OPENBAO_ADDR/v1/auth/approle/login" \
      -H 'Content-Type: application/json' \
      -d "{\"role_id\":\"$OPENBAO_ROLE_ID\",\"secret_id\":\"$OPENBAO_SECRET_ID\"}" \
      | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

    git clone http://heezy-admin:${{ secrets.GIT_CLONE_TOKEN }}@gitea.heezy.local:3000/heezy-admin/terraform-heezy.git /tmp/terraform-heezy
    cd /tmp/terraform-heezy && git checkout ${{ github.sha }}

    cd environments/<env>/<provider>
    terraform init && terraform plan -out=tfplan && terraform apply -auto-approve tfplan

Known Issues¶

`gitea-db-sync` is suspended¶

Runs can appear stuck as "running", or land on conclusion: cancelled while every step succeeded. The gitea-db-sync CronJob in the gitea namespace fixes that, but it is currently suspended and last ran 12 days ago, so nothing self-heals. Check job logs and step-level status for ground truth.

kubectl get cronjob gitea-db-sync -n gitea
kubectl patch cronjob gitea-db-sync -n gitea -p '{"spec":{"suspend":false}}'

Manual fix:

UPDATE action_run_job SET status=3 WHERE status IN (1,2)
  AND task_id IN (SELECT id FROM action_task WHERE stopped>0);

DinD image cache is ephemeral¶

DinD storage is emptyDir — all pulled layers are lost on pod restart. Build steps use docker pull :latest || true plus --cache-from :latest to pull the previous image as a cache source and skip unchanged pip/apt layers.

`failed-pod-cleaner`¶

A CronJob in the gitea namespace reaps failed job pods every 15 minutes. If job pods are piling up in Error, check that it is running before hunting elsewhere.