homelab/k8s-ng/core/prom-stack

Monitoring stack (new cluster)

The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log chunks, or Grafana database are copied from the old cluster.

Layout

• kube-prometheus-stack: Prometheus, Alertmanager, Grafana, exporters and Kubernetes alerting rules.
• loki: one filesystem-backed monolithic instance. A second replica would require shared object storage and is deliberately not faked with a local PV.
• alloy: unprivileged Kubernetes API log and event collection. This replaces EOL Promtail and does not require host log mounts or privileged sysctls.
• Grafana state is stored in the main CNPG cluster; metrics and logs stay on the large local disk of node music.

Storage and retention

The PreSync hook creates these directories on music:

Path	PV	Retention
/data/k8s/monitoring/prometheus	500 Gi	up to 365 days / 425 GB
/data/k8s/monitoring/loki	200 Gi	90 days
/data/k8s/monitoring/alertmanager	5 Gi	120 hours of notification state

All PVs and the StorageClass use Retain; Argo CD is also told not to prune them. This protects data from a manifest rename, but it is not a backup. Back up /data/k8s/monitoring with the rest of the music node if history matters.

Dependencies

• node music must exist and /data must be the intended large filesystem;
• main-rw.psql.svc and the generated grafana-db Secret must be ready;
• vaultwarden-fields must resolve k8s/prometheus/grafana-secrets with the custom fields IDM Client ID, IDM Client Secret, TG Token, and TG ID;
• DNS for gf.hexor.cy and prom.hexor.cy must point to the new cluster.

After Argo CD reports the application healthy, verify targets, rules and log delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana uses its own Keycloak OIDC client.