Skip to content

Monitoring

The monitoring stack runs on big-boi (192.168.1.21) as Docker Compose services managed by the lgtm Ansible role. Config and data live under /mnt/lgtm, on big-boi's 916G second disk rather than its 114G root volume: Prometheus alone is 25G at 15d retention.

Last verified: 2026-08-25.

Stack

Service Port URL Purpose
Grafana 3000 https://grafana.heezy.info Dashboards, alerting
Prometheus 9090 http://192.168.1.21:9090 Metrics collection, 15d retention
Loki 3100 — Log aggregation, 3d retention
Tempo 3200 — Traces. OTLP on 4317/4318, Zipkin on 9411, Jaeger on 14268
Mimir 9009 — Long-term metrics storage
snmp-exporter 9116 — Cisco switch, FortiGate and Ruckus AP metrics
Promtail 514/UDP — Syslog receiver for network gear

prometheus.heezy.info does not exist

Grafana is the only component with a DNS name; grafana.heezy.info → 192.168.1.25 (SWAG). There is no prometheus.heezy.info record in heezy-info-overrides.j2, so the Prometheus UI is only reachable at http://192.168.1.21:9090. Adding the record is one line in the dnsmasq role plus a SWAG proxy conf, if that is wanted.

Dashboards

All dashboards are provisioned from code in ansible-heezy/roles/lgtm/files/dashboards/. Grafana loads them at startup via file provisioning — any dashboard created manually in the UI is deleted on the next Ansible run.

To add or modify a dashboard: edit the JSON in files/dashboards/, commit, push to gitea, and let playbook-lgtm-execution.yml deploy it.

Five dashboards ship today:

File Dashboard
k8s_cluster.json heezy — K8s Cluster
node_exporter.json heezy — Node Exporter
media_pipeline.json heezy — Media Pipeline
capacity_overview.json heezy — Capacity Overview
network_overview.json heezy — Network Overview

heezy — K8s Cluster (k8s_cluster.json)

  • Cluster Overview — node count, pod counts, restart count, CPU/memory %, PVC health
  • Node Resources — per-node CPU and memory time series, top-15 pod CPU and memory
  • Pod Table — all pods with phase, restart count, CPU, memory
  • Network — node Rx/Tx throughput, pod Rx/Tx top-10, Cisco switch port utilization and errors
  • Storage — PVC usage bar gauge and table with used/capacity/percent
  • Container Restarts — restart rate over time

Variables: namespace (default: heezy), node

heezy — Node Exporter (node_exporter.json)

  • Host Overview — uptime, CPU cores, RAM, load average, root disk free, open FDs
  • CPU — usage %, mode breakdown, system load, context switches
  • Memory — used/available/buffers/cached, swap
  • Disk — space usage bar gauge, I/O throughput, latency, IOPS
  • Network — throughput, errors/drops, TCP connections, socket stats

Variable: instance (multi-select, all hosts)

heezy — Media Pipeline (media_pipeline.json)

  • Soulseek / slskd — connection status, VPN status, downloads PVC usage, reconnect attempt log, slskd logs filtered to errors/downloads/searches
  • Lidarr — error/grab logs
  • Soularr — activity logs

Datasources

Provisioned from grafana-datasources.yml.j2:

Name Type Points at
Prometheus prometheus http://prometheus:9090
Loki loki http://loki:3100
Tempo tempo http://tempo:3200
Mimir prometheus http://mimir:9009/prometheus
PostgreSQL - heezy postgres bigboi.heezy.local:5432

Prometheus Scrape Targets

There is no Prometheus Operator and no ServiceMonitor CRDs. Every job is a static target rendered by Ansible from inventory/hosts.yml, in roles/lgtm/templates/prometheus.yml.j2. Adding a target is an ansible-heezy change, not a k8s manifest.

Job Targets Interval
node-exporter every host in the linux group, :9100 15s
promtail every host in the linux group, :9080 15s
windows-exporter the windows group, :9182 — empty today 15s
kubelet nebula-1..5 :10255 /metrics 15s
kubelet-cadvisor nebula-1..5 :10255 /metrics/cadvisor 15s
kube-state-metrics nebula1.heezy.local:30800 (NodePort — no VIP exists for it) 15s
nvidia-gpu-exporter bigboi.heezy.local:9835 15s
snmp cisco-switch, fortigate, ruckus-ap (if_mib) and ups (ups_mib), via snmp-exporter 30s
prometheus / loki / grafana / tempo / mimir localhost 15s

Only the Ruckus master AP answers SNMP; ruckus-ap-member has no agent and is deliberately not polled.

The fortigate SNMP target is down

snmp/fortigate.heezy.local fails with context deadline exceeded. The FortiGate is reachable and answering; the heezy-ro SNMP community's host list is an allow-list of query sources and still names 192.168.1.10, so a poll from big-boi gets no reply at all rather than a refusal, which is why it reads as a timeout instead of an auth failure.

The fix is one line in terraform-heezy/environments/production/heezy/snmp.tf. FortiGate interface metrics and the Network Overview dashboard's FortiGate panels are empty until it lands. cisco-switch, ups and ruckus-ap are unaffected.

To list every unhealthy target:

curl -s 'http://192.168.1.21:9090/api/v1/targets?state=active' \
  | python3 -c "import sys,json;[print(t['labels'].get('job'),t['labels'].get('instance'),t['health']) for t in json.load(sys.stdin)['data']['activeTargets'] if t['health']!='up']"

The textfile collector

node_exporter runs with --collector.textfile.directory=/var/lib/node_exporter/textfile_collectors on every Linux host, so any host can publish its own metrics by dropping a .prom file there. Only big-boi uses it today, for the steam-headless GLX probe.

The unit is owned solely by the prometheus-node-exporter role. Nothing else may write /etc/systemd/system/node_exporter.service: a second writer that omits the flag silently disables every textfile metric on that host, and the metrics disappear rather than going to zero.

Alerts

All alert rules are in ansible-heezy/roles/lgtm/templates/grafana-alerting.yml.j2, all in the heezy-prometheus folder. Contact points and notification policies come from grafana-contactpoints.yml.j2 and grafana-policies.yml.j2. Notifications go to Discord.

Alert Groups

Group Rules
host-alerts CPU Usage High, High Memory Usage, Host Down
disk-alerts Low Root Disk Space, Disk Space Critical, Node Root Disk Usage High, Node Root Disk Filling Fast
switch-alerts Switch Port Errors, Switch Port High Utilization
k8s-alerts Container Restart Spike, PVC Lost or Stuck Pending
k8s-job-alerts Scheduled Job Failed — any CronJob in heezy that exhausts its backoffLimit. Runbook
storage-alerts slskd Downloads PVC High Usage, slskd Downloads PVC Critical, slskd Disconnected from Soulseek
gameserver-alerts CS 1.6 Player Joined, UT2K4 Player Joined, slskd Unauthorized API Key, slskd Soulseek Auth Failure, slskd VPN Failure, qBittorrent VPN Failure
statement-parser-alerts Statement Parser Error
oauth-alerts Gmail OAuth Token Expired. Runbook
steam-headless-alerts Steam Headless Not Rendering On NVIDIA, Steam Headless GLX Probe Stale

There is no pvc-alerts group. Longhorn PVC fullness is not alerted on generically — only the slskd downloads PVC has thresholds, under storage-alerts. A cluster-wide PVC usage alert is a gap.

Idempotency

The lgtm role is fully idempotent:

  • Dashboard JSONs are copied from files/dashboards/ — unmanaged JSONs are deleted
  • Alert rules, contact points and notification policies are provisioned from templates
  • Datasources are provisioned from grafana-datasources.yml.j2
  • All config is regenerated on every run; Grafana restarts if any config changes

Runbook: Grafana Shows No Data

  1. Prometheus healthy: curl http://192.168.1.21:9090/-/healthy
  2. Scrape targets: http://192.168.1.21:9090/targets
  3. node-exporter on a host: curl http://192.168.1.15:9100/metrics | head
  4. kube-state-metrics: curl http://192.168.1.15:30800/metrics | head
  5. If a datasource is broken, re-run playbook-lgtm-execution.yml from Gitea

Runbook: Add a New Dashboard

  1. Build the dashboard JSON in the Grafana UI
  2. Export via Grafana → Dashboard → Export → Save to file
  3. Strip the id field, set a stable uid
  4. Place it in ansible-heezy/roles/lgtm/files/dashboards/<name>.json
  5. Commit and push to gitea — the lgtm playbook deploys it on the next run

Remember step 4. A dashboard that only exists in the UI is deleted on the next run.