fix(observability): reap orphans, bound tasks, cap the TSDB (#71)

README.md

@@ -45,7 +45,7 @@ incident.
   can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
   identically everywhere. [How](docs/architecture.md#observability-data-flow)
 - **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and
-  51 alert rules — 38 metric-based in Prometheus, 13 log-based in Loki — sharing
+  52 alert rules — 39 metric-based in Prometheus, 13 log-based in Loki — sharing
   one Alertmanager routing tree. No dashboard exists only in a database.
 - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
   decrypted at deploy time into gitignored paths, with `git log` showing which
@@ -138,7 +138,7 @@ rack; a dashed border means egress only. Full topology and data flow in
 | --- | --- | --- |
 | Firewall / routing | [pfSense on FreeBSD 15](docs/network.md) | VLANs, DHCP, default-deny |
 | Virtualisation | Proxmox VE | Lab hypervisor |
-| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention, remote-write receiver |
+| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention capped at 12 GiB, remote-write receiver |
 | Logs | [Loki](stacks/observability/loki) | Single-binary, filesystem storage |
 | Collection | [Grafana Alloy](stacks/observability/alloy) | node + cAdvisor metrics, Docker/journal/syslog/auth logs |
 | Network polling | [snmp_exporter](stacks/observability/snmp-exporter) | pfSense, switch, UPS, iLO |
@@ -153,7 +153,7 @@ rack; a dashed border means egress only. Full topology and data flow in
 .
 ├── stacks/observability/     # the deployed stack — one compose file, six services
 │   ├── compose.yaml
-│   ├── prometheus/           # config, file_sd targets, 38 alert rules
+│   ├── prometheus/           # config, file_sd targets, 39 alert rules
 │   ├── alertmanager/         # routing and inhibition
 │   ├── loki/                 # single-binary config + 13 LogQL rules
 │   ├── alloy/                # one agent config, used on every host

docs/observability.md

@@ -22,6 +22,15 @@ Retention is 30 days for both metrics (`PROMETHEUS_RETENTION` in `.env`) and
 logs (`retention_period` in `loki/loki-config.yaml`). Change both together or
 dashboards will show metrics with no matching logs at the far end of the range.
 
+Metrics carry a second bound: `PROMETHEUS_RETENTION_SIZE` caps the store at
+12 GiB, and whichever limit is reached first wins. It is sized from the write
+rate rather than the store's current size — 72.9 MiB/day gives 2.28 GiB over
+30 days, 2.73 GiB once `Saruman` gets an agent — so it is roughly 4.4x the
+planned estate and should never bind. In normal operation the 30 days above is
+the limit that applies; if the size cap ever binds, that 30 days stops being
+true and `PrometheusSizeRetentionActive` is what says so. Loki has no
+equivalent size bound.
+
 ## Log level normalisation
 
 Logs arrive spelling severity about twenty different ways — `ERROR`, `err`,
@@ -70,7 +79,7 @@ expression in every panel is syntactically valid.
 
 ## Alerting
 
-51 rules in total: 38 metric-based in `prometheus/rules/`, and 13 log-based in
+52 rules in total: 39 metric-based in `prometheus/rules/`, and 13 log-based in
 `loki/rules/`.
 
 ### Log-based (Loki ruler)
@@ -99,7 +108,7 @@ boot check.
 
 ### Metric-based (Prometheus)
 
-38 rules across six files in `prometheus/rules/`:
+39 rules across six files in `prometheus/rules/`:
 
 | File | Covers |
 | --- | --- |
@@ -117,7 +126,7 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho
 `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
 feed a rule synthetic series and assert it fires — paired with a case asserting
 it stays quiet, because a test that only ever expects silence would have passed
-against the broken rule too. Coverage is two rules of 35 so far:
+against the broken rule too. Coverage is six rules of 39 so far:
 `ContainerHighMemory` and `Watchdog`. The other 33 are still validated for
 syntax only, which is exactly the standing #63 had.
 

docs/roadmap.md

@@ -54,12 +54,21 @@ issues intact. Nothing was summarised away.
   something you find out about from a browser warning.
 - **[#114](https://github.com/Gerrrt/HomeLab/issues/114) Set memory limits on
   the six services.** Nothing in `compose.yaml` bounds a leak, so one container
-  can take the host down — and the host has 8 GB soldered. Blocked on data
-  rather than on agreement: cAdvisor has only reported correctly since
-  [#62](https://github.com/Gerrrt/HomeLab/pull/62), so there are hours of
-  history to size from, not the 30 days retention implies, and `grafana` alone
-  swings 3.7x inside that window. A limit picked from it would be a guess at an
-  OOM kill.
+  can take the host down — and the host has 8 GB soldered. It was blocked on
+  data: cAdvisor has only reported correctly since
+  [#62](https://github.com/Gerrrt/HomeLab/pull/62), and `grafana` swung 3.7x
+  inside the six hours available, so a limit picked from it would have been a
+  guess at an OOM kill. Nine more days did not settle it — they widened it.
+  `loki` now swings 8.6x (121 MiB median, 1039 MiB peak), and `loki`, `grafana`
+  and `alloy` all peaked in the *same hour* on 2026-08-29, which is an episodic
+  event rather than a distribution that converges with more sampling. The
+  method the issue proposed no longer fits the machine either: 3x every peak is
+  7536 MiB against 7816 MiB of RAM. So the gate has changed rather than moved —
+  waiting for more history is not what unblocks this, explaining that one hour
+  is. [#71](https://github.com/Gerrrt/HomeLab/issues/71) took the half that was
+  sizeable: `pids_limit`, because tens of threads against a 10,000-thread abort
+  is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which
+  is at a measurable steady state at day 28 of 30.
 - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard
   screenshots.** `make screenshots` does four of the five; the Logs dashboard is
   deliberately excluded. → [`images/README.md`](images/README.md)

docs/runbooks/deploy-stack.md

@@ -63,11 +63,48 @@ ss -ltn | grep -E ':(9090|3100|3000)'    # BIND_ADDR — 0.0.0.0 by default
 ss -lun | grep ':1514'                   # BIND_ADDR
 ```
 
+Confirm the per-container limits applied too. A limit that silently failed to
+take is the thing they exist to defend against, and nothing in the stack would
+report it (#71):
+
+```bash
+for c in prometheus alertmanager loki grafana snmp-exporter blackbox-exporter alloy; do
+  docker inspect "$c" --format '{{.Name}} init={{.HostConfig.Init}} pids={{.HostConfig.PidsLimit}}'
+done
+```
+
+Every service should read `init=true`, and `pids=512` except `alloy`, which is
+`1024`. That reads back what Docker was *asked* for, so confirm the kernel
+agrees — a stale cgroup directory will happily show you the old ceiling:
+
+```bash
+p=$(docker inspect -f '{{.State.Pid}}' prometheus)
+cat "/sys/fs/cgroup$(cut -d: -f3 /proc/$p/cgroup | head -1)/pids.max"    # 512
+```
+
+`init=false` on grafana means the zombie leak is back:
+
+```bash
+docker exec grafana sh -c 'ls -d /proc/[0-9]* | wc -l'
+```
+
+which should be single digits and stay there, not climb by two every 30 seconds.
+
+Confirm the size ceiling is armed, too. `PrometheusSizeRetentionActive` only
+fires once the ceiling *bites*, so if the flag were ever dropped this would
+silently read `0` and the rule would stay quiet forever — the #63 shape:
+
+```bash
+curl -s 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_retention_limit_bytes'
+```
+
+`12884901888` is 12 GiB. `0` means there is no size bound in force.
+
 Then in the UI:
 
 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
    take up to 45 seconds on their first scrape.
-2. **Prometheus → Status → Rules.** 38 rules loaded, none in error.
+2. **Prometheus → Status → Rules.** 39 rules loaded, none in error.
 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated.
 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving.
 5. Confirm level normalisation is working — this has been silently broken

docs/security.md

@@ -14,7 +14,7 @@ What this network is actually built to survive:
 | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only |
 | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
 | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
-| Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs |
+| Losing visibility of a failure | 39 alert rules, 30 days of metrics and logs |
 | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
 | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below |
 
@@ -243,6 +243,13 @@ complete state table and interface topology. They are credentials.
   [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md).
 - The Alloy debug UI binds to `127.0.0.1` only.
 - The Docker socket is mounted read-only into Alloy.
+- Every service runs under a real init (`init: true`) and a chosen task ceiling
+  (`pids_limit`, 512; 1024 for Alloy) rather than the inherited systemd default
+  of 9056. This was not theoretical: Grafana's https healthcheck was leaking two
+  unreaped `ssl_client` children every 30 seconds and would have exhausted the
+  inherited ceiling about three days after each start (#71).
+- Prometheus carries a byte ceiling as well as a time one, so a change that
+  quietly multiplies the series count cannot consume the disk unnoticed.
 - All images are pinned to explicit versions, so an upstream compromise cannot
   arrive silently via `:latest`. Dependabot proposes the bumps; CI validates
   them.

stacks/observability/.env.example

@@ -40,3 +40,13 @@ SYSLOG_PORT=1514
 # Metric retention. Loki's log retention is set separately in
 # loki/loki-config.yaml (retention_period) — keep the two in step.
 PROMETHEUS_RETENTION=30d
+
+# A byte ceiling on the same store, so whichever limit is reached first wins.
+# Sized from the per-block write rate, not from the store's current size: at
+# 72.9 MiB/day, 30 days is 2.28 GiB, or 2.73 GiB once Saruman gets an agent
+# (#88). 12 GiB is 4.4x that and 12% of the volume. It is meant never to bind —
+# if it does, retention is quietly shorter than the line above claims, and
+# PrometheusSizeRetentionActive is what says so. compose.yaml has the full
+# derivation and the reason a smaller number would have been wrong. Loki has no
+# size bound at all.
+PROMETHEUS_RETENTION_SIZE=12GB

stacks/observability/README.md

@@ -32,7 +32,7 @@ prometheus/
   prometheus.yaml          scrape config; SNMP via file_sd
   targets/snmp.yaml        SNMP targets — hot-reloaded, no restart needed
   targets/blackbox.yaml    probe targets — hot-reloaded, no restart needed
-  rules/*.rules.yaml       38 alert rules across host/network/ups/containers/blackbox
+  rules/*.rules.yaml       39 alert rules across host/network/ups/containers/blackbox
   tests/*.test.yaml        promtool unit tests — assert the rules can fire
 blackbox/blackbox.yaml     probe modules — reachability from outside the service
 alertmanager/

stacks/observability/compose.yaml

@@ -20,6 +20,44 @@ x-service-defaults: &service-defaults
   restart: unless-stopped
   logging: *default-logging
   networks: [observability]
+  # A real init as PID 1, so orphaned children are reaped. Not hygiene: on
+  # 2026-08-31 grafana was holding 2437 defunct `ssl_client` processes and
+  # gaining two more every 30 seconds. Busybox wget forks a TLS helper for the
+  # https healthcheck further down, the helper is reparented to PID 1 when wget
+  # exits, and PID 1 in that container is `grafana server` — which never calls
+  # wait(). The default pids.max is 9056, so it ran out of tasks about three
+  # days after each start, and was only ever saved by being redeployed sooner
+  # than that. Every other healthcheck here is plain HTTP and forks nothing,
+  # which is why grafana was the only one that showed it (#71).
+  init: true
+  # Bounds a task leak of the kind the paragraph above describes. Steady state
+  # is 9-17 tasks per service and the highest go_threads seen over nine days is
+  # 21, so 512 is ~25x normal and still well under the Go runtime's own
+  # 10,000-thread abort. The number is loose on purpose: what makes a PID limit
+  # safe to set — and a memory limit not, which is why there is no mem_limit in
+  # this file and #114 owns that question — is that the gap between "certainly
+  # enough" and "certainly protective" spans two orders of magnitude here. A
+  # container that hits this dies on EAGAIN, restart: unless-stopped recovers
+  # it, and ContainerRestartLoop fires. The quieter case is the one that
+  # actually happened: the app was fine and the *healthcheck* could not fork,
+  # which shows as `unhealthy` and is not something Docker restarts — the same
+  # gap this file notes on loki. There is no rule watching PID counts directly
+  # because cAdvisor is not exporting container_processes; Alloy's exporter
+  # disables the "process" metric kind, and turning it on means restating the
+  # whole disabled list, since enabled_metrics overrides it rather than adding
+  # to it. Getting that wrong drops every other container metric, which is the
+  # #62 shape exactly. Worth its own issue, not a line here.
+  #
+  # `ulimits` is absent, and deliberately. nofile would throttle the mmap'd
+  # chunk files prometheus and loki depend on, to defend against an exhaustion
+  # never observed here. nproc is worse than useless: RLIMIT_NPROC is enforced
+  # per real UID across the whole kernel, not per container, and alertmanager
+  # and snmp-exporter both run as ${RENDER_UID} — the operator's own uid, set
+  # from `id -u` by scripts/render-config.sh. An nproc there would count the
+  # operator's host processes against the container, and could refuse the
+  # operator a login shell. pids_limit is a cgroup control and is scoped
+  # correctly; ulimits: nproc is a uid control in a container costume.
+  pids_limit: 512
 
 services:
   # ---------------------------------------------------------------------------
@@ -35,6 +73,43 @@ services:
       - --config.file=/etc/prometheus/prometheus.yaml
       - --storage.tsdb.path=/prometheus
       - --storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-30d}
+      # A ceiling on the TSDB, not a tidy-up. It bounds *sustained* growth —
+      # the series count creeping up as hosts and exporters are added — so that
+      # the disk cannot be consumed a few MiB at a time by a change nobody
+      # thought of as a storage decision.
+      #
+      # Do not size it from the store's current size. On 2026-08-31 that was
+      # 1021 MiB, and prometheus_tsdb_time_retentions_total was 0 — nothing had
+      # ever been evicted, so the store was still filling and that number was
+      # not a steady state. Per-block rates over the retained window: 16.1
+      # MiB/day up to 08-17, then 72.9 MiB/day after, because #62 fixed cAdvisor
+      # and the series count went 3,514 -> 24,005. Steady state at the current
+      # rate is 30 x 72.9 + ~150 MiB of WAL and head chunks = 2.28 GiB, and
+      # ~2.73 GiB once Saruman gets an agent (#88). So 12GB — powers-of-2, i.e.
+      # 12 GiB — is 5.3x today and 4.4x with #88, and 12% of the 98 GiB volume.
+      # 4 GiB looked generous against 1021 MiB and would have been 1.5x against
+      # the estate that is already planned.
+      #
+      # That headroom is the point: this repository promises 30 days in four
+      # places, and a ceiling reached in normal operation is a second retention
+      # policy wearing a ceiling's clothes. If it ever does evict,
+      # PrometheusSizeRetentionActive in containers.rules.yaml says so.
+      #
+      # What it does NOT do is win a race against a flood through the
+      # remote-write receiver enabled below, which anything that can route here
+      # may push to (ADR-0012, #182). The budget counts the WAL and the m-mapped
+      # head chunks but only ever evicts persistent blocks, and a flood lands in
+      # the WAL: at a few MB/s the head alone passes 12 GiB long before any
+      # block does, at which point every block is deleted and the WAL keeps
+      # growing regardless. Bounding that is authentication's job, not this
+      # flag's.
+      #
+      # Both retention flags are marked [DEPRECATED] by this version in favour
+      # of a storage.tsdb.retention block in prometheus.yaml. They still work,
+      # and they stay here because prometheus.yaml is mounted rather than
+      # rendered — moving retention into it would drop these two tunables out of
+      # .env.example, which is where this repository deliberately keeps them.
+      - --storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE:-12GB}
       - --web.enable-remote-write-receiver
       - --web.enable-lifecycle
     volumes:
@@ -321,6 +396,10 @@ services:
     # from the environment instead; the header of that file has the whole story.
     image: grafana/alloy:v1.19.2@sha256:b8ec653c44235fbe910879145dac3597d66b0aaecf60bcbbe82580767771a839
     container_name: alloy
+    # 1024 rather than the shared 512: this is the one service that walks
+    # cgroups, tails the journal and talks to the Docker socket, so it has more
+    # reason than the others to fan out. Observed peak is 20 threads either way.
+    pids_limit: 1024
     privileged: true
     # Without this the cAdvisor exporter reports exactly one series, id="/",
     # and every per-container metric is missing — which is what the Docker
@@ -414,6 +493,13 @@ services:
     # through a large dashboard, which presents as a truncated PNG rather than
     # an error.
     shm_size: "1gb"
+    # Overrides the shared 512, which is a number measured from Go services and
+    # means nothing here — a headless Chromium is a process tree, not one
+    # binary. This value is not measured either: the `capture` profile is off on
+    # every normal deploy, so there was nothing running to count. It is set
+    # rather than inherited so that the shared default cannot quietly break a
+    # `make screenshots` run, and it is high enough that it should not bind.
+    pids_limit: 4096
 
   # ---------------------------------------------------------------------------
   # tar and gzip for scripts/backup-volumes.sh. NOT part of the running stack: