# Monitoring stack (new cluster)

The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log
chunks, or Grafana database are copied from the old cluster.

## Layout

- `kube-prometheus-stack`: Prometheus, Alertmanager, Grafana, exporters and
  Kubernetes alerting rules.
- `loki`: one filesystem-backed monolithic instance. A second replica would
  require shared object storage and is deliberately not faked with a local PV.
- `alloy`: unprivileged Kubernetes API log and event collection. This replaces
  EOL Promtail and does not require host log mounts or privileged sysctls.
- Grafana state is stored in the `main` CNPG cluster; metrics and logs stay on
  the large local disk of node `music`.

## Storage and retention

The PreSync hook creates these directories on `music`:

| Path | PV | Retention |
| --- | ---: | --- |
| `/data/k8s/monitoring/prometheus` | 500 Gi | up to 365 days / 425 GB |
| `/data/k8s/monitoring/loki` | 200 Gi | 90 days |
| `/data/k8s/monitoring/alertmanager` | 5 Gi | 120 hours of notification state |

All PVs and the StorageClass use `Retain`; Argo CD is also told not to prune
them. This protects data from a manifest rename, but it is not a backup. Back up
`/data/k8s/monitoring` with the rest of the `music` node if history matters.

## Dependencies

- node `music` must exist and `/data` must be the intended large filesystem;
- `main-rw.psql.svc` and the generated `grafana-db` Secret must be ready;
- `vaultwarden-login` must resolve the existing Grafana OIDC and Telegram item
  UUIDs used in `external-secrets.yaml`;
- DNS for `gf.hexor.cy` and `prom.hexor.cy` must point to the new cluster.

After Argo CD reports the application healthy, verify targets, rules and log
delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana
uses its own Keycloak OIDC client.