Added prom-stack

k8s-ng/core/prom-stack/README.md

@@ -0,0 +1,41 @@
+# Monitoring stack (new cluster)
+
+The stack is intentionally a fresh installation: no Prometheus TSDB, Loki log
+chunks, or Grafana database are copied from the old cluster.
+
+## Layout
+
+- `kube-prometheus-stack`: Prometheus, Alertmanager, Grafana, exporters and
+  Kubernetes alerting rules.
+- `loki`: one filesystem-backed monolithic instance. A second replica would
+  require shared object storage and is deliberately not faked with a local PV.
+- `alloy`: unprivileged Kubernetes API log and event collection. This replaces
+  EOL Promtail and does not require host log mounts or privileged sysctls.
+- Grafana state is stored in the `main` CNPG cluster; metrics and logs stay on
+  the large local disk of node `music`.
+
+## Storage and retention
+
+The PreSync hook creates these directories on `music`:
+
+| Path | PV | Retention |
+| --- | ---: | --- |
+| `/data/k8s/monitoring/prometheus` | 500 Gi | up to 365 days / 425 GB |
+| `/data/k8s/monitoring/loki` | 200 Gi | 90 days |
+| `/data/k8s/monitoring/alertmanager` | 5 Gi | 120 hours of notification state |
+
+All PVs and the StorageClass use `Retain`; Argo CD is also told not to prune
+them. This protects data from a manifest rename, but it is not a backup. Back up
+`/data/k8s/monitoring` with the rest of the `music` node if history matters.
+
+## Dependencies
+
+- node `music` must exist and `/data` must be the intended large filesystem;
+- `main-rw.psql.svc` and the generated `grafana-db` Secret must be ready;
+- `vaultwarden-login` must resolve the existing Grafana OIDC and Telegram item
+  UUIDs used in `external-secrets.yaml`;
+- DNS for `gf.hexor.cy` and `prom.hexor.cy` must point to the new cluster.
+
+After Argo CD reports the application healthy, verify targets, rules and log
+delivery in Grafana. Prometheus is protected by the Keycloak auth proxy; Grafana
+uses its own Keycloak OIDC client.