Skip to content

Monitoring

The monitoring stack runs on shared-lgtm (192.168.1.10) as Docker Compose services managed by the lgtm Ansible role.

Stack

Service Port URL Purpose
Grafana 3000 grafana.heezy.info Dashboards, alerting
Prometheus 9090 prometheus.heezy.info Metrics collection
Loki 3100 — Log aggregation
Tempo 3200 — Distributed tracing
Mimir 9009 — Long-term metrics storage
SNMP Exporter 9116 — Cisco switch metrics

Dashboards

All dashboards are provisioned from code in ansible-heezy/roles/lgtm/files/dashboards/. Grafana loads them at startup via file provisioning — any dashboard created manually in the UI will be deleted on the next Ansible run.

To add or modify a dashboard: edit the JSON in files/dashboards/, commit, push to gitea, and let the lgtm playbook deploy it.

heezy — K8s Cluster (k8s_cluster.json)

Comprehensive Kubernetes cluster view. Sections:

  • Cluster Overview — node count, pod counts, restart count, CPU/memory %, PVC health
  • Node Resources — per-node CPU and memory time series, top-15 pod CPU and memory
  • Pod Table — all pods with phase, restart count, CPU, memory
  • Network — node Rx/Tx throughput, pod Rx/Tx top-10, Cisco switch port utilization and errors (SNMP)
  • Storage — PVC usage bar gauge and table with used/capacity/percent
  • Container Restarts — restart rate over time

Variables: namespace (default: heezy), node

heezy — Node Exporter (node_exporter.json)

Per-host Linux metrics. Sections:

  • Host Overview — uptime, CPU cores, RAM, load average, root disk free, open FDs
  • CPU — usage %, mode breakdown, system load, context switches
  • Memory — used/available/buffers/cached, swap
  • Disk — space usage bar gauge, I/O throughput, latency, IOPS
  • Network — throughput, errors/drops, TCP connections, socket stats

Variable: instance (multi-select, all hosts)

heezy — Media Pipeline (media_pipeline.json)

End-to-end view of the music acquisition pipeline. Sections:

  • Soulseek / slskd — connection status, VPN status, downloads PVC usage (gauge + time series), reconnect attempt log, slskd logs filtered to errors/downloads/searches
  • Lidarr — error/grab logs
  • Soularr — activity logs (all output)

Prometheus Scrape Targets

Job Targets Interval
node-exporter All Linux hosts :9100 15s
kubelet-cadvisor nebula-1..5 :10250/metrics/cadvisor 15s
kube-state-metrics nebula-1:30800 15s
nvidia-gpu-exporter big-boi:9835 15s
snmp cisco-switch (192.168.1.26) 30s
prometheus / loki / grafana / tempo / mimir localhost 15s

Alerts

All alert rules are in ansible-heezy/roles/lgtm/templates/grafana-alerting.yml.j2. Notifications go to Discord via webhook.

Alert Groups

Group Rules
host-alerts CPU > 80% (30m), Memory > 90% (5m), Host Down (1m)
disk-alerts Root disk < 10% (5m), Root disk < 5% (2m)
pvc-alerts PVC > 85% (5m), PVC > 95% (2m)
storage-alerts slskd downloads PVC > 80% (10m), > 95% (2m), slskd disconnected 30m
switch-alerts Port errors > 1/s (5m), Port utilization > 80% (20m)
k8s-alerts Container restart spike > 3 in 30m
k8s-job-alerts Scheduled Job Failed: any CronJob in heezy that exhausts its backoffLimit. Runbook
gameserver-alerts CS 1.6 player joined, UT2K4 player joined, slskd API key rejected, slskd auth failure, slskd VPN failure, qBittorrent VPN failure
statement-parser-alerts Statement parser error

Idempotency

The lgtm Ansible role is fully idempotent:

  • Dashboard JSONs are copied from files/dashboards/ — unmanaged JSONs are deleted
  • Alert rules, contact points, and notification policies are provisioned from templates
  • Datasources are provisioned from grafana-datasources.yml.j2
  • All config is regenerated on every run; Grafana restarts if any config changes

Runbook: Grafana Shows No Data

  1. Check Prometheus is up: curl http://192.168.1.10:9090/-/healthy
  2. Check scrape targets: http://prometheus.heezy.info/targets
  3. Check node-exporter on a host: curl http://192.168.1.15:9100/metrics | head
  4. Check kube-state-metrics: curl http://192.168.1.15:30800/metrics | head
  5. If Grafana datasource is broken: re-run playbook-lgtm-execution.yml from Gitea

Runbook: Add a New Dashboard

  1. Build the dashboard JSON in Grafana UI (use allowUiUpdates: true in provisioning)
  2. Export via Grafana → Dashboard → Export → Save to file
  3. Strip id field, set a stable uid
  4. Place in ansible-heezy/roles/lgtm/files/dashboards/<name>.json
  5. Commit and push to gitea — the lgtm playbook deploys it on next run