Aleksandr Bogomiakov
9b0530fb70
Update Kubernetes Services Wiki / Generate and Update K8s Wiki (push) Successful in 9s
64 lines
3.1 KiB
Markdown
64 lines
3.1 KiB
Markdown
# Monitoring stack (new cluster)
|
|
|
|
The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log
|
|
chunks, or Grafana database are copied from the old cluster.
|
|
|
|
## Layout
|
|
|
|
- `kube-prometheus-stack`: Prometheus, Alertmanager, Grafana, exporters and
|
|
Kubernetes alerting rules.
|
|
- `loki`: one filesystem-backed monolithic instance. A second replica would
|
|
require shared object storage and is deliberately not faked with a local PV.
|
|
- `alloy`: unprivileged Kubernetes API log and event collection. This replaces
|
|
EOL Promtail and does not require host log mounts or privileged sysctls.
|
|
- Grafana state is stored in the `main` CNPG cluster; metrics and logs stay on
|
|
the large local disk of node `music`.
|
|
|
|
Stack services select `homelab.hexor.cy/pool=core` and
|
|
`homelab.hexor.cy/exposure=external`. Prometheus, Alertmanager and Loki also
|
|
select `music` because their volumes are local. Grafana, the operators, Alloy
|
|
and the Loki gateway can run on either `master` or `music`. Node Exporter and
|
|
Loki Canary are not restricted to these groups.
|
|
|
|
## Storage and retention
|
|
|
|
The PreSync hook creates these directories on `music`:
|
|
|
|
| Path | PV | Retention |
|
|
| --- | ---: | --- |
|
|
| `/data/k8s/monitoring/prometheus` | 500 Gi | up to 365 days / 425 GB |
|
|
| `/data/k8s/monitoring/loki` | 200 Gi | 90 days |
|
|
| `/data/k8s/monitoring/alertmanager` | 5 Gi | 120 hours of notification state |
|
|
|
|
All PVs and the StorageClass use `Retain`; Argo CD is also told not to prune
|
|
them. This protects data from a manifest rename, but it is not a backup. Back up
|
|
`/data/k8s/monitoring` with the rest of the `music` node if history matters.
|
|
|
|
## Dependencies
|
|
|
|
- node `music` must exist and `/data` must be the intended large filesystem;
|
|
- `main-rw.psql.svc` and the generated `grafana-db` Secret must be ready;
|
|
- `vaultwarden-fields` must resolve `k8s/prometheus/grafana-secrets` with the
|
|
custom fields `IDM Client ID`, `IDM Client Secret`, `TG Token`, and `TG ID`;
|
|
- DNS for `gf.hexor.cy` and `prom.hexor.cy` must point to the new cluster.
|
|
|
|
## Alerting and target identities
|
|
|
|
Telegram is reserved for actionable alerts: `severity=info`, `Watchdog`, and
|
|
`InfoInhibitor` remain visible in Prometheus but route to the null receiver.
|
|
Warning and critical notifications are grouped for two minutes and rendered as
|
|
Telegram HTML with escaped annotations and labelled alerts/runbook links.
|
|
Notifications show at most three alerts, with each alert's escaped text limited
|
|
to 500 characters. A single short Prometheus alerts link replaces repeated graph
|
|
URLs; the full group remains visible there. Workload alerts show the affected
|
|
DaemonSet, Deployment or StatefulSet rather than the kube-state-metrics pod
|
|
which supplied the metric.
|
|
|
|
Node Exporter keeps scraping the discovered node address, but rewrites the
|
|
public `instance` label from Kubernetes discovery to `<node-name>:9100`. Grafana
|
|
variables and alerts therefore remain stable when node IP addresses change.
|
|
|
|
After Argo CD reports the application healthy, verify targets, rules and log
|
|
delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana
|
|
uses its own Keycloak OIDC client.
|