# Monitoring stack (new cluster)

The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log
chunks, or Grafana database are copied from the old cluster.

## Layout

- `kube-prometheus-stack`: Prometheus, Alertmanager, Grafana, exporters and
  Kubernetes alerting rules.
- `loki`: one filesystem-backed monolithic instance. A second replica would
  require shared object storage and is deliberately not faked with a local PV.
- `alloy`: unprivileged Kubernetes API log and event collection. This replaces
  EOL Promtail and does not require host log mounts or privileged sysctls.
- Grafana state is stored in the `main` CNPG cluster; metrics and logs stay on
  the large local disk of node `music`.

Stack services select `homelab.hexor.cy/pool=core` and
`homelab.hexor.cy/exposure=external`. Prometheus, Alertmanager and Loki also
select `music` because their volumes are local. Grafana, the operators, Alloy
and the Loki gateway can run on either `master` or `music`. Node Exporter and
Loki Canary are not restricted to these groups.

## Storage and retention

The PreSync hook creates these directories on `music`:

| Path | PV | Retention |
| --- | ---: | --- |
| `/data/k8s/monitoring/prometheus` | 500 Gi | up to 365 days / 425 GB |
| `/data/k8s/monitoring/loki` | 200 Gi | 90 days |
| `/data/k8s/monitoring/alertmanager` | 5 Gi | 120 hours of notification state |

All PVs and the StorageClass use `Retain`; Argo CD is also told not to prune
them. This protects data from a manifest rename, but it is not a backup. Back up
`/data/k8s/monitoring` with the rest of the `music` node if history matters.

## Dependencies

- node `music` must exist and `/data` must be the intended large filesystem;
- `main-rw.psql.svc` and the generated `grafana-db` Secret must be ready;
- `vaultwarden-fields` must resolve `k8s/prometheus/grafana-secrets` with the
  custom fields `IDM Client ID`, `IDM Client Secret`, `TG Token`, and `TG ID`;
- DNS for `gf.hexor.cy` and `prom.hexor.cy` must point to the new cluster.

## Alerting and target identities

Telegram is reserved for actionable alerts: `severity=info`, `Watchdog`, and
`InfoInhibitor` remain visible in Prometheus but route to the null receiver.
Warning and critical notifications are grouped for two minutes and rendered as
Telegram HTML with escaped annotations and labelled alerts/runbook links.
Notifications show at most three alerts, with each alert's escaped text limited
to 500 characters. A single short Prometheus alerts link replaces repeated graph
URLs; the full group remains visible there. Workload alerts show the affected
DaemonSet, Deployment or StatefulSet rather than the kube-state-metrics pod
which supplied the metric.

Node Exporter keeps scraping the discovered node address, but rewrites the
public `instance` label from Kubernetes discovery to `<node-name>:9100`. Grafana
variables and alerts therefore remain stable when node IP addresses change.

After Argo CD reports the application healthy, verify targets, rules and log
delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana
uses its own Keycloak OIDC client.