diff --git a/README.md b/README.md index de6a425..b0de7ce 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and - 51 alert rules — 38 metric-based in Prometheus, 13 log-based in Loki — sharing + 52 alert rules — 39 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -138,7 +138,7 @@ rack; a dashed border means egress only. Full topology and data flow in | --- | --- | --- | | Firewall / routing | [pfSense on FreeBSD 15](docs/network.md) | VLANs, DHCP, default-deny | | Virtualisation | Proxmox VE | Lab hypervisor | -| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention, remote-write receiver | +| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention capped at 12 GiB, remote-write receiver | | Logs | [Loki](stacks/observability/loki) | Single-binary, filesystem storage | | Collection | [Grafana Alloy](stacks/observability/alloy) | node + cAdvisor metrics, Docker/journal/syslog/auth logs | | Network polling | [snmp_exporter](stacks/observability/snmp-exporter) | pfSense, switch, UPS, iLO | @@ -153,7 +153,7 @@ rack; a dashed border means egress only. Full topology and data flow in . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 38 alert rules +│ ├── prometheus/ # config, file_sd targets, 39 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # one agent config, used on every host diff --git a/docs/observability.md b/docs/observability.md index aedc223..ff61501 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -22,6 +22,15 @@ Retention is 30 days for both metrics (`PROMETHEUS_RETENTION` in `.env`) and logs (`retention_period` in `loki/loki-config.yaml`). Change both together or dashboards will show metrics with no matching logs at the far end of the range. +Metrics carry a second bound: `PROMETHEUS_RETENTION_SIZE` caps the store at +12 GiB, and whichever limit is reached first wins. It is sized from the write +rate rather than the store's current size — 72.9 MiB/day gives 2.28 GiB over +30 days, 2.73 GiB once `Saruman` gets an agent — so it is roughly 4.4x the +planned estate and should never bind. In normal operation the 30 days above is +the limit that applies; if the size cap ever binds, that 30 days stops being +true and `PrometheusSizeRetentionActive` is what says so. Loki has no +equivalent size bound. + ## Log level normalisation Logs arrive spelling severity about twenty different ways — `ERROR`, `err`, @@ -70,7 +79,7 @@ expression in every panel is syntactically valid. ## Alerting -51 rules in total: 38 metric-based in `prometheus/rules/`, and 13 log-based in +52 rules in total: 39 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -99,7 +108,7 @@ boot check. ### Metric-based (Prometheus) -38 rules across six files in `prometheus/rules/`: +39 rules across six files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -117,7 +126,7 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is two rules of 35 so far: +against the broken rule too. Coverage is six rules of 39 so far: `ContainerHighMemory` and `Watchdog`. The other 33 are still validated for syntax only, which is exactly the standing #63 had. diff --git a/docs/roadmap.md b/docs/roadmap.md index af59aaf..7546766 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -54,12 +54,21 @@ issues intact. Nothing was summarised away. something you find out about from a browser warning. - **[#114](https://github.com/Gerrrt/HomeLab/issues/114) Set memory limits on the six services.** Nothing in `compose.yaml` bounds a leak, so one container - can take the host down — and the host has 8 GB soldered. Blocked on data - rather than on agreement: cAdvisor has only reported correctly since - [#62](https://github.com/Gerrrt/HomeLab/pull/62), so there are hours of - history to size from, not the 30 days retention implies, and `grafana` alone - swings 3.7x inside that window. A limit picked from it would be a guess at an - OOM kill. + can take the host down — and the host has 8 GB soldered. It was blocked on + data: cAdvisor has only reported correctly since + [#62](https://github.com/Gerrrt/HomeLab/pull/62), and `grafana` swung 3.7x + inside the six hours available, so a limit picked from it would have been a + guess at an OOM kill. Nine more days did not settle it — they widened it. + `loki` now swings 8.6x (121 MiB median, 1039 MiB peak), and `loki`, `grafana` + and `alloy` all peaked in the *same hour* on 2026-08-29, which is an episodic + event rather than a distribution that converges with more sampling. The + method the issue proposed no longer fits the machine either: 3x every peak is + 7536 MiB against 7816 MiB of RAM. So the gate has changed rather than moved — + waiting for more history is not what unblocks this, explaining that one hour + is. [#71](https://github.com/Gerrrt/HomeLab/issues/71) took the half that was + sizeable: `pids_limit`, because tens of threads against a 10,000-thread abort + is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which + is at a measurable steady state at day 28 of 30. - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard screenshots.** `make screenshots` does four of the five; the Logs dashboard is deliberately excluded. → [`images/README.md`](images/README.md) diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index bba9779..5f9792d 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -63,11 +63,48 @@ ss -ltn | grep -E ':(9090|3100|3000)' # BIND_ADDR — 0.0.0.0 by default ss -lun | grep ':1514' # BIND_ADDR ``` +Confirm the per-container limits applied too. A limit that silently failed to +take is the thing they exist to defend against, and nothing in the stack would +report it (#71): + +```bash +for c in prometheus alertmanager loki grafana snmp-exporter blackbox-exporter alloy; do + docker inspect "$c" --format '{{.Name}} init={{.HostConfig.Init}} pids={{.HostConfig.PidsLimit}}' +done +``` + +Every service should read `init=true`, and `pids=512` except `alloy`, which is +`1024`. That reads back what Docker was *asked* for, so confirm the kernel +agrees — a stale cgroup directory will happily show you the old ceiling: + +```bash +p=$(docker inspect -f '{{.State.Pid}}' prometheus) +cat "/sys/fs/cgroup$(cut -d: -f3 /proc/$p/cgroup | head -1)/pids.max" # 512 +``` + +`init=false` on grafana means the zombie leak is back: + +```bash +docker exec grafana sh -c 'ls -d /proc/[0-9]* | wc -l' +``` + +which should be single digits and stay there, not climb by two every 30 seconds. + +Confirm the size ceiling is armed, too. `PrometheusSizeRetentionActive` only +fires once the ceiling *bites*, so if the flag were ever dropped this would +silently read `0` and the rule would stay quiet forever — the #63 shape: + +```bash +curl -s 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_retention_limit_bytes' +``` + +`12884901888` is 12 GiB. `0` means there is no size bound in force. + Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 38 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 39 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/security.md b/docs/security.md index 2b52868..5b4b803 100644 --- a/docs/security.md +++ b/docs/security.md @@ -14,7 +14,7 @@ What this network is actually built to survive: | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only | | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | -| Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 39 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | @@ -243,6 +243,13 @@ complete state table and interface topology. They are credentials. [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). - The Alloy debug UI binds to `127.0.0.1` only. - The Docker socket is mounted read-only into Alloy. +- Every service runs under a real init (`init: true`) and a chosen task ceiling + (`pids_limit`, 512; 1024 for Alloy) rather than the inherited systemd default + of 9056. This was not theoretical: Grafana's https healthcheck was leaking two + unreaped `ssl_client` children every 30 seconds and would have exhausted the + inherited ceiling about three days after each start (#71). +- Prometheus carries a byte ceiling as well as a time one, so a change that + quietly multiplies the series count cannot consume the disk unnoticed. - All images are pinned to explicit versions, so an upstream compromise cannot arrive silently via `:latest`. Dependabot proposes the bumps; CI validates them. diff --git a/stacks/observability/.env.example b/stacks/observability/.env.example index a1ee1e5..e5e4109 100644 --- a/stacks/observability/.env.example +++ b/stacks/observability/.env.example @@ -40,3 +40,13 @@ SYSLOG_PORT=1514 # Metric retention. Loki's log retention is set separately in # loki/loki-config.yaml (retention_period) — keep the two in step. PROMETHEUS_RETENTION=30d + +# A byte ceiling on the same store, so whichever limit is reached first wins. +# Sized from the per-block write rate, not from the store's current size: at +# 72.9 MiB/day, 30 days is 2.28 GiB, or 2.73 GiB once Saruman gets an agent +# (#88). 12 GiB is 4.4x that and 12% of the volume. It is meant never to bind — +# if it does, retention is quietly shorter than the line above claims, and +# PrometheusSizeRetentionActive is what says so. compose.yaml has the full +# derivation and the reason a smaller number would have been wrong. Loki has no +# size bound at all. +PROMETHEUS_RETENTION_SIZE=12GB diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 05e401d..356f7ba 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox.yaml probe targets — hot-reloaded, no restart needed - rules/*.rules.yaml 38 alert rules across host/network/ups/containers/blackbox + rules/*.rules.yaml 39 alert rules across host/network/ups/containers/blackbox tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability from outside the service alertmanager/ diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 125217e..0710df9 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -20,6 +20,44 @@ x-service-defaults: &service-defaults restart: unless-stopped logging: *default-logging networks: [observability] + # A real init as PID 1, so orphaned children are reaped. Not hygiene: on + # 2026-08-31 grafana was holding 2437 defunct `ssl_client` processes and + # gaining two more every 30 seconds. Busybox wget forks a TLS helper for the + # https healthcheck further down, the helper is reparented to PID 1 when wget + # exits, and PID 1 in that container is `grafana server` — which never calls + # wait(). The default pids.max is 9056, so it ran out of tasks about three + # days after each start, and was only ever saved by being redeployed sooner + # than that. Every other healthcheck here is plain HTTP and forks nothing, + # which is why grafana was the only one that showed it (#71). + init: true + # Bounds a task leak of the kind the paragraph above describes. Steady state + # is 9-17 tasks per service and the highest go_threads seen over nine days is + # 21, so 512 is ~25x normal and still well under the Go runtime's own + # 10,000-thread abort. The number is loose on purpose: what makes a PID limit + # safe to set — and a memory limit not, which is why there is no mem_limit in + # this file and #114 owns that question — is that the gap between "certainly + # enough" and "certainly protective" spans two orders of magnitude here. A + # container that hits this dies on EAGAIN, restart: unless-stopped recovers + # it, and ContainerRestartLoop fires. The quieter case is the one that + # actually happened: the app was fine and the *healthcheck* could not fork, + # which shows as `unhealthy` and is not something Docker restarts — the same + # gap this file notes on loki. There is no rule watching PID counts directly + # because cAdvisor is not exporting container_processes; Alloy's exporter + # disables the "process" metric kind, and turning it on means restating the + # whole disabled list, since enabled_metrics overrides it rather than adding + # to it. Getting that wrong drops every other container metric, which is the + # #62 shape exactly. Worth its own issue, not a line here. + # + # `ulimits` is absent, and deliberately. nofile would throttle the mmap'd + # chunk files prometheus and loki depend on, to defend against an exhaustion + # never observed here. nproc is worse than useless: RLIMIT_NPROC is enforced + # per real UID across the whole kernel, not per container, and alertmanager + # and snmp-exporter both run as ${RENDER_UID} — the operator's own uid, set + # from `id -u` by scripts/render-config.sh. An nproc there would count the + # operator's host processes against the container, and could refuse the + # operator a login shell. pids_limit is a cgroup control and is scoped + # correctly; ulimits: nproc is a uid control in a container costume. + pids_limit: 512 services: # --------------------------------------------------------------------------- @@ -35,6 +73,43 @@ services: - --config.file=/etc/prometheus/prometheus.yaml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-30d} + # A ceiling on the TSDB, not a tidy-up. It bounds *sustained* growth — + # the series count creeping up as hosts and exporters are added — so that + # the disk cannot be consumed a few MiB at a time by a change nobody + # thought of as a storage decision. + # + # Do not size it from the store's current size. On 2026-08-31 that was + # 1021 MiB, and prometheus_tsdb_time_retentions_total was 0 — nothing had + # ever been evicted, so the store was still filling and that number was + # not a steady state. Per-block rates over the retained window: 16.1 + # MiB/day up to 08-17, then 72.9 MiB/day after, because #62 fixed cAdvisor + # and the series count went 3,514 -> 24,005. Steady state at the current + # rate is 30 x 72.9 + ~150 MiB of WAL and head chunks = 2.28 GiB, and + # ~2.73 GiB once Saruman gets an agent (#88). So 12GB — powers-of-2, i.e. + # 12 GiB — is 5.3x today and 4.4x with #88, and 12% of the 98 GiB volume. + # 4 GiB looked generous against 1021 MiB and would have been 1.5x against + # the estate that is already planned. + # + # That headroom is the point: this repository promises 30 days in four + # places, and a ceiling reached in normal operation is a second retention + # policy wearing a ceiling's clothes. If it ever does evict, + # PrometheusSizeRetentionActive in containers.rules.yaml says so. + # + # What it does NOT do is win a race against a flood through the + # remote-write receiver enabled below, which anything that can route here + # may push to (ADR-0012, #182). The budget counts the WAL and the m-mapped + # head chunks but only ever evicts persistent blocks, and a flood lands in + # the WAL: at a few MB/s the head alone passes 12 GiB long before any + # block does, at which point every block is deleted and the WAL keeps + # growing regardless. Bounding that is authentication's job, not this + # flag's. + # + # Both retention flags are marked [DEPRECATED] by this version in favour + # of a storage.tsdb.retention block in prometheus.yaml. They still work, + # and they stay here because prometheus.yaml is mounted rather than + # rendered — moving retention into it would drop these two tunables out of + # .env.example, which is where this repository deliberately keeps them. + - --storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE:-12GB} - --web.enable-remote-write-receiver - --web.enable-lifecycle volumes: @@ -321,6 +396,10 @@ services: # from the environment instead; the header of that file has the whole story. image: grafana/alloy:v1.19.2@sha256:b8ec653c44235fbe910879145dac3597d66b0aaecf60bcbbe82580767771a839 container_name: alloy + # 1024 rather than the shared 512: this is the one service that walks + # cgroups, tails the journal and talks to the Docker socket, so it has more + # reason than the others to fan out. Observed peak is 20 threads either way. + pids_limit: 1024 privileged: true # Without this the cAdvisor exporter reports exactly one series, id="/", # and every per-container metric is missing — which is what the Docker @@ -414,6 +493,13 @@ services: # through a large dashboard, which presents as a truncated PNG rather than # an error. shm_size: "1gb" + # Overrides the shared 512, which is a number measured from Go services and + # means nothing here — a headless Chromium is a process tree, not one + # binary. This value is not measured either: the `capture` profile is off on + # every normal deploy, so there was nothing running to count. It is set + # rather than inherited so that the shared default cannot quietly break a + # `make screenshots` run, and it is high enough that it should not bind. + pids_limit: 4096 # --------------------------------------------------------------------------- # tar and gzip for scripts/backup-volumes.sh. NOT part of the running stack: diff --git a/stacks/observability/prometheus/rules/containers.rules.yaml b/stacks/observability/prometheus/rules/containers.rules.yaml index 234497b..a7fa2f0 100644 --- a/stacks/observability/prometheus/rules/containers.rules.yaml +++ b/stacks/observability/prometheus/rules/containers.rules.yaml @@ -89,6 +89,32 @@ groups: annotations: summary: "{{ $value }} rule evaluation failures in the last 15 minutes" + - alert: PrometheusSizeRetentionActive + # --storage.tsdb.retention.size is a ceiling that, when it engages, + # silently shortens retention from the 30 days everything else in this + # repository claims to whatever fits in 4 GiB. Nothing else would say + # so: the store keeps working, the dashboards keep drawing, and you find + # out when a query over last month comes back short. Prometheus counts + # size-based eviction separately from time-based eviction, which is the + # only thing that distinguishes "the ceiling bound" from "old data aged + # out normally" — do not substitute prometheus_tsdb_time_retentions_total + # here, it increments constantly and by design. + # + # Firing means one of two things and both want a human: the series + # budget grew past what 4 GiB holds for 30 days, or something is pushing + # volume into the open remote-write receiver. Either way the documented + # retention is no longer true. + expr: increase(prometheus_tsdb_size_retentions_total[1h]) > 0 + labels: + component: containers + severity: warning + category: capacity + annotations: + summary: "Prometheus is evicting blocks to stay under its size ceiling" + description: >- + Retention is now bounded by PROMETHEUS_RETENTION_SIZE rather than + PROMETHEUS_RETENTION, so history is shorter than 30 days. + - alert: AlertmanagerNotificationsFailing expr: increase(alertmanager_notifications_failed_total[15m]) > 0 for: 5m diff --git a/stacks/observability/prometheus/tests/containers.test.yaml b/stacks/observability/prometheus/tests/containers.test.yaml index b08b860..5f968b5 100644 --- a/stacks/observability/prometheus/tests/containers.test.yaml +++ b/stacks/observability/prometheus/tests/containers.test.yaml @@ -148,3 +148,45 @@ tests: category: capacity exp_annotations: summary: "Container alloy is using 50% of host memory" + + # --- PrometheusSizeRetentionActive: fires when the ceiling engages -------- + # The counter is flat until the size ceiling starts evicting blocks, then it + # increments once per eviction. One step inside the hour is the whole signal. + - interval: 1m + input_series: + - series: 'prometheus_tsdb_size_retentions_total{instance="localhost:9090",job="prometheus"}' + values: "0+0x40 1+0x30" + alert_rule_test: + - eval_time: 60m + alertname: PrometheusSizeRetentionActive + exp_alerts: + - exp_labels: + alertname: PrometheusSizeRetentionActive + instance: localhost:9090 + job: prometheus + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Prometheus is evicting blocks to stay under its size ceiling" + description: >- + Retention is now bounded by PROMETHEUS_RETENTION_SIZE rather + than PROMETHEUS_RETENTION, so history is shorter than 30 days. + + # --- PrometheusSizeRetentionActive: quiet while the ceiling is not bound -- + # This is the production state and the one the rule must not alert on: the + # counter exists and reads 0 forever. Without this case the rule would pass + # its test while being wired to a metric that is always absent. + - interval: 1m + input_series: + - series: 'prometheus_tsdb_size_retentions_total{instance="localhost:9090",job="prometheus"}' + values: "0x70" + # Time-based eviction runs constantly and by design. If the expression is + # ever "simplified" onto this counter instead, the rule fires every hour + # forever, and this case is what catches that. + - series: 'prometheus_tsdb_time_retentions_total{instance="localhost:9090",job="prometheus"}' + values: "0+1x70" + alert_rule_test: + - eval_time: 60m + alertname: PrometheusSizeRetentionActive + exp_alerts: []