2026-10-07 18:35:17 +01:00
|
|
|
# Monitoring stack (new cluster)
|
|
|
|
|
|
|
|
|
|
The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log
|
|
|
|
|
chunks, or Grafana database are copied from the old cluster.
|
|
|
|
|
|
|
|
|
|
## Layout
|
|
|
|
|
|
|
|
|
|
- `kube-prometheus-stack`: Prometheus, Alertmanager, Grafana, exporters and
|
|
|
|
|
Kubernetes alerting rules.
|
|
|
|
|
- `loki`: one filesystem-backed monolithic instance. A second replica would
|
|
|
|
|
require shared object storage and is deliberately not faked with a local PV.
|
|
|
|
|
- `alloy`: unprivileged Kubernetes API log and event collection. This replaces
|
|
|
|
|
EOL Promtail and does not require host log mounts or privileged sysctls.
|
|
|
|
|
- Grafana state is stored in the `main` CNPG cluster; metrics and logs stay on
|
|
|
|
|
the large local disk of node `music`.
|
|
|
|
|
|
|
|
|
|
## Storage and retention
|
|
|
|
|
|
|
|
|
|
The PreSync hook creates these directories on `music`:
|
|
|
|
|
|
|
|
|
|
| Path | PV | Retention |
|
|
|
|
|
| --- | ---: | --- |
|
|
|
|
|
| `/data/k8s/monitoring/prometheus` | 500 Gi | up to 365 days / 425 GB |
|
|
|
|
|
| `/data/k8s/monitoring/loki` | 200 Gi | 90 days |
|
|
|
|
|
| `/data/k8s/monitoring/alertmanager` | 5 Gi | 120 hours of notification state |
|
|
|
|
|
|
|
|
|
|
All PVs and the StorageClass use `Retain`; Argo CD is also told not to prune
|
|
|
|
|
them. This protects data from a manifest rename, but it is not a backup. Back up
|
|
|
|
|
`/data/k8s/monitoring` with the rest of the `music` node if history matters.
|
|
|
|
|
|
|
|
|
|
## Dependencies
|
|
|
|
|
|
|
|
|
|
- node `music` must exist and `/data` must be the intended large filesystem;
|
|
|
|
|
- `main-rw.psql.svc` and the generated `grafana-db` Secret must be ready;
|
2026-10-07 18:52:06 +01:00
|
|
|
- `vaultwarden-fields` must resolve `k8s/prometheus/grafana-secrets` with the
|
|
|
|
|
custom fields `IDM Client ID`, `IDM Client Secret`, `TG Token`, and `TG ID`;
|
2026-10-07 18:35:17 +01:00
|
|
|
- DNS for `gf.hexor.cy` and `prom.hexor.cy` must point to the new cluster.
|
|
|
|
|
|
2026-10-08 17:03:46 +01:00
|
|
|
## Alerting and target identities
|
|
|
|
|
|
|
|
|
|
Telegram is reserved for actionable alerts: `severity=info`, `Watchdog`, and
|
|
|
|
|
`InfoInhibitor` remain visible in Prometheus but route to the null receiver.
|
|
|
|
|
Warning and critical notifications are grouped for two minutes and rendered as
|
|
|
|
|
Telegram HTML with escaped annotations and labelled graph/runbook links.
|
|
|
|
|
|
|
|
|
|
Node Exporter keeps scraping the discovered node address, but rewrites the
|
|
|
|
|
public `instance` label from Kubernetes discovery to `<node-name>:9100`. Grafana
|
|
|
|
|
variables and alerts therefore remain stable when node IP addresses change.
|
|
|
|
|
|
2026-10-07 18:35:17 +01:00
|
|
|
After Argo CD reports the application healthy, verify targets, rules and log
|
|
|
|
|
delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana
|
|
|
|
|
uses its own Keycloak OIDC client.
|
