CI/CD Runners¶

Gitea act-runners running in Kubernetes handle all CI/CD. GitHub Actions are not used — GitHub is a read-only mirror.

Quick Reference¶

Setting	Value
Gitea URL (internal)	`http://192.168.1.31:3000` (MetalLB VIP)
Gitea URL (for DinD jobs)	`http://192.168.1.15:30360` (NodePort — DinD can't resolve k8s DNS)
`DOCKER_HOST` (build steps)	`tcp://172.17.0.1:2375`
Runner labels	`ubuntu-latest`, `ubuntu-22.04`, `self-hosted`
Job container image	`025066240222.dkr.ecr.us-east-2.amazonaws.com/heezy-runner:latest`
Active replicas	3 pods × 2 concurrent jobs = 6 total
Runner namespace	`gitea`

Architecture¶

  Gitea (http://192.168.1.31:3000)
       |
       | act-runner registration
       |
  ┌────▼─────────────────────────────────────────────┐
  │  K8s Deployment: act-runner (gitea namespace)    │
  │  3 replicas, spread across nebula-1..5           │
  │                                                  │
  │  Pod (×3):                                       │
  │  ┌──────────────────┐  ┌──────────────────────┐  │
  │  │  act_runner       │  │  docker:27-dind       │  │
  │  │                   │◄─┤  (privileged)         │  │
  │  │                   │  │  DOCKER_HOST exposed  │  │
  │  │                   │  │  at tcp://172.17.0.1  │  │
  │  └──────────────────┘  └──────────────────────┘  │
  └──────────────────────────────────────────────────┘
       |
       | job container pulled from ECR
       |
  ┌────▼──────────────────────────────────────────────┐
  │  heezy-runner:latest (ECR)                        │
  │  Contains: ansible, terraform, aws cli, kubectl,  │
  │  docker cli, git, python3, jq, sshpass            │
  └───────────────────────────────────────────────────┘

heezy-runner Image¶

All runner labels resolve to a custom ECR image built in ansible-heezy:

Image: 025066240222.dkr.ecr.us-east-2.amazonaws.com/heezy-runner:latest
Built by: build-heezy-runner.yml workflow in ansible-heezy
Contains: ansible-core + collections, terraform 1.12.2, aws CLI, docker CLI, kubectl, git, python3, jq, sshpass
No app code baked in — repos are always cloned at runtime from Gitea

This replaced the old ansible-automation image + catthehacker/ubuntu:act-22.04 approach.

ECR Pull Auth¶

Runner pods need ECR credentials to pull the job image:

ecr-refresh-aws-creds secret — AWS creds for ECR login (created manually once)
ecr-credentials secret — docker-registry secret, refreshed every 6h by ecr-credentials-refresh CronJob
imagePullSecrets in act-runner deployment references ecr-credentials

Workflow Authoring Rules¶

Push to Gitea, not GitHub¶

# Always push Gitea first — GitHub does NOT trigger workflows
git push gitea main && git push origin main

VAULT_TOKEN must not cross step boundaries¶

The act-runner redacts masked values written to $GITHUB_ENV. Never do:

# BROKEN — next step reads VAULT_TOKEN=***
echo "VAULT_TOKEN=$VAULT_TOKEN" >> $GITHUB_ENV

Fetch and use the token in the same shell step.

DOCKER_HOST required for docker commands¶

Job steps run directly in heezy-runner. Steps that call docker build/docker push must set:

env:
  DOCKER_HOST: tcp://172.17.0.1:2375

Ansible and terraform steps do NOT need this.

Workflow files location¶

All workflows go in .gitea/workflows/ only. Never use .github/workflows/ — the act-runner executes both, causing duplicate runs.

Shell compatibility¶

Act-runner uses /bin/sh, not bash. Replace all [[ ]] constructs:

# BROKEN in sh
EMOJI=$([[ "$STATUS" == "success" ]] && echo "✅" || echo "❌")

# Works in sh
if [ "$STATUS" = "success" ]; then EMOJI="✅"; else EMOJI="❌"; fi

Git clone target¶

Always clone from internal Gitea NodePort (DinD can't use k8s DNS):

git clone http://192.168.1.15:30360/heezy-admin/<repo>.git

Ansible Workflow Pattern¶

- name: Run playbook
  env:
    OPENBAO_ADDR: http://openbao.heezy.local:8200
    OPENBAO_ROLE_ID: ${{ secrets.OPENBAO_ROLE_ID }}
    OPENBAO_SECRET_ID: ${{ secrets.OPENBAO_SECRET_ID }}
    ANSIBLE_HOST_KEY_CHECKING: "False"
  run: |
    VAULT_TOKEN=$(curl -sk -X POST "$OPENBAO_ADDR/v1/auth/approle/login" \
      -H 'Content-Type: application/json' \
      -d "{\"role_id\":\"$OPENBAO_ROLE_ID\",\"secret_id\":\"$OPENBAO_SECRET_ID\"}" \
      | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

    git clone http://192.168.1.15:30360/heezy-admin/ansible-heezy.git /tmp/ansible-heezy
    cd /tmp/ansible-heezy && git checkout ${{ github.sha }}

    ansible-playbook -i inventory/hosts.yml playbooks/<name>.yml

Terraform Workflow Pattern¶

- name: Terraform apply
  env:
    OPENBAO_ADDR: http://openbao.heezy.local:8200
    OPENBAO_ROLE_ID: ${{ secrets.OPENBAO_ROLE_ID }}
    OPENBAO_SECRET_ID: ${{ secrets.OPENBAO_SECRET_ID }}
  run: |
    VAULT_TOKEN=$(curl -sk -X POST "$OPENBAO_ADDR/v1/auth/approle/login" \
      -H 'Content-Type: application/json' \
      -d "{\"role_id\":\"$OPENBAO_ROLE_ID\",\"secret_id\":\"$OPENBAO_SECRET_ID\"}" \
      | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

    git clone http://192.168.1.15:30360/heezy-admin/terraform-heezy.git /tmp/terraform-heezy
    cd /tmp/terraform-heezy && git checkout ${{ github.sha }}

    cd environments/<env>/heezy
    terraform init && terraform plan -out=tfplan && terraform apply -auto-approve tfplan

Known Issues¶

Gitea DB sync lag¶

Runs can appear stuck as "running" in the UI. gitea-db-sync CronJob (gitea namespace) patches run status every 5 minutes. UI may lag up to 5 min — check job logs for ground truth.

Manual fix:

UPDATE action_run_job SET status=3 WHERE status IN (1,2)
  AND task_id IN (SELECT id FROM action_task WHERE stopped>0);

DinD image cache is ephemeral¶

DinD storage is emptyDir — all pulled layers are lost on pod restart. ECR --cache-from :latest is used in build steps to mitigate this.