homelab/k8s-ng/core/prom-stack/README.md

Monitoring stack (new cluster)

The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log chunks, or Grafana database are copied from the old cluster.

Layout

• kube-prometheus-stack: Prometheus, Alertmanager, Grafana, exporters and Kubernetes alerting rules.
• loki: one filesystem-backed monolithic instance. A second replica would require shared object storage and is deliberately not faked with a local PV.
• alloy: unprivileged Kubernetes API log and event collection. This replaces EOL Promtail and does not require host log mounts or privileged sysctls.
• Grafana state is stored in the main CNPG cluster; metrics and logs stay on the large local disk of node music.

Stack services select homelab.hexor.cy/pool=core and homelab.hexor.cy/exposure=external. Prometheus, Alertmanager and Loki also select music because their volumes are local. Grafana, the operators, Alloy and the Loki gateway can run on either master or music. Node Exporter and Loki Canary are not restricted to these groups.

Storage and retention

The PreSync hook creates these directories on music:

Path	PV	Retention
/data/k8s/monitoring/prometheus	500 Gi	up to 365 days / 425 GB
/data/k8s/monitoring/loki	200 Gi	90 days
/data/k8s/monitoring/alertmanager	5 Gi	120 hours of notification state

All PVs and the StorageClass use Retain; Argo CD is also told not to prune them. This protects data from a manifest rename, but it is not a backup. Back up /data/k8s/monitoring with the rest of the music node if history matters.

Dependencies

• node music must exist and /data must be the intended large filesystem;
• main-rw.psql.svc and the generated grafana-db Secret must be ready;
• vaultwarden-fields must resolve k8s/prometheus/grafana-secrets with the custom fields IDM Client ID, IDM Client Secret, TG Token, and TG ID;
• DNS for gf.hexor.cy and prom.hexor.cy must point to the new cluster.

Alerting and target identities

Telegram is reserved for actionable alerts: severity=info, Watchdog, and InfoInhibitor remain visible in Prometheus but route to the null receiver. Warning and critical notifications are grouped for two minutes and rendered as Telegram HTML with escaped annotations and labelled alerts/runbook links. Notifications show at most three alerts, with each alert's escaped text limited to 500 characters. A single short Prometheus alerts link replaces repeated graph URLs; the full group remains visible there. Workload alerts show the affected DaemonSet, Deployment or StatefulSet rather than the kube-state-metrics pod which supplied the metric.

Node Exporter keeps scraping the discovered node address, but rewrites the public instance label from Kubernetes discovery to <node-name>:9100. Grafana variables and alerts therefore remain stable when node IP addresses change.

After Argo CD reports the application healthy, verify targets, rules and log delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana uses its own Keycloak OIDC client.