Aleksandr Bogomiakov
52bfa88a33
Update Kubernetes Services Wiki / Generate and Update K8s Wiki (push) Successful in 31s
42 lines
1.8 KiB
Markdown
42 lines
1.8 KiB
Markdown
# Monitoring stack (new cluster)
|
|
|
|
The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log
|
|
chunks, or Grafana database are copied from the old cluster.
|
|
|
|
## Layout
|
|
|
|
- `kube-prometheus-stack`: Prometheus, Alertmanager, Grafana, exporters and
|
|
Kubernetes alerting rules.
|
|
- `loki`: one filesystem-backed monolithic instance. A second replica would
|
|
require shared object storage and is deliberately not faked with a local PV.
|
|
- `alloy`: unprivileged Kubernetes API log and event collection. This replaces
|
|
EOL Promtail and does not require host log mounts or privileged sysctls.
|
|
- Grafana state is stored in the `main` CNPG cluster; metrics and logs stay on
|
|
the large local disk of node `music`.
|
|
|
|
## Storage and retention
|
|
|
|
The PreSync hook creates these directories on `music`:
|
|
|
|
| Path | PV | Retention |
|
|
| --- | ---: | --- |
|
|
| `/data/k8s/monitoring/prometheus` | 500 Gi | up to 365 days / 425 GB |
|
|
| `/data/k8s/monitoring/loki` | 200 Gi | 90 days |
|
|
| `/data/k8s/monitoring/alertmanager` | 5 Gi | 120 hours of notification state |
|
|
|
|
All PVs and the StorageClass use `Retain`; Argo CD is also told not to prune
|
|
them. This protects data from a manifest rename, but it is not a backup. Back up
|
|
`/data/k8s/monitoring` with the rest of the `music` node if history matters.
|
|
|
|
## Dependencies
|
|
|
|
- node `music` must exist and `/data` must be the intended large filesystem;
|
|
- `main-rw.psql.svc` and the generated `grafana-db` Secret must be ready;
|
|
- `vaultwarden-login` must resolve the existing Grafana OIDC and Telegram item
|
|
UUIDs used in `external-secrets.yaml`;
|
|
- DNS for `gf.hexor.cy` and `prom.hexor.cy` must point to the new cluster.
|
|
|
|
After Argo CD reports the application healthy, verify targets, rules and log
|
|
delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana
|
|
uses its own Keycloak OIDC client.
|