fix(observability): reap orphans, bound tasks, cap the TSDB (#71)

[Gerrrt]

Closes the half of #71 that could be sized. The memory half stays with #114.

What #71 asked for, and what the data said

#71 lists mem_limit, cpus, deploy.resources, pids_limit and ulimits as
all absent, plus the missing --storage.tsdb.retention.size. Two of those turn
out to be sizeable today and the rest do not, which is the shape of this PR.

Memory is not sizeable, and nine more days made that clearer rather than
less. #114 was blocked on getting 14 days of container history; there are now
9.4, and the distribution widened:

service	median	peak	swing
loki	121 MiB	1039 MiB	8.6x
grafana	359 MiB	630 MiB	1.8x
alloy	285 MiB	512 MiB	1.8x

loki, grafana and alloy all peaked in the same hour on 2026-08-29 —
episodic, not a distribution that converges with sampling. And #114's own
method no longer fits the machine: 3x every peak is 7536 MiB against 7816 MiB
of RAM. So docs/roadmap.md now says the gate has changed rather than moved —
what unblocks #114 is explaining that one hour, not waiting for more history.

The leak this found

Verifying #71 turned up something already running. Grafana was holding 2437
defunct ssl_client processes, gaining two more every 30 seconds:

pids.current = 2466, of which 2437 are Z, ppid=1
+2 every 30s — exactly the healthcheck interval

BusyBox wget forks a TLS helper for the https healthcheck; on exit the helper
is reparented to PID 1, which here is grafana server and never calls wait().
Only Grafana was affected — every other healthcheck is plain HTTP and forks
nothing. At 2880 charges/day against the inherited pids.max of 9056 that is a
fork failure about 3.1 days after each start, presenting as unhealthy,
which Docker does not restart. It has only ever been saved by being redeployed
sooner than that.

Nothing in the stack could see it: container_tasks_state reads zero for every
state and node_processes_state only sees Alloy's own namespace. Same shape as
#63 — a fault that runs healthy and produces nothing.

init: true therefore comes first and pids_limit second. The reverse order
would have cut Grafana off sooner rather than protecting it.

Changes

• init: true on the shared anchor, so every service has a reaping PID 1.
• pids_limit: 512 (1024 alloy, 4096 for the unmeasured renderer) against
  an observed 9–17 tasks and a 21-thread peak. Loose on purpose: what makes a
  task ceiling safe to set and a memory limit not is that the gap between
  "certainly enough" and "certainly protective" spans two orders of magnitude.
  ulimits stays absent and the file records why — RLIMIT_NPROC is enforced
  per real UID kernel-wide, and alertmanager/snmp-exporter run as the
  operator's own uid via RENDER_UID.
• --storage.tsdb.retention.size=12GB, plumbed through
  PROMETHEUS_RETENTION_SIZE.
• PrometheusSizeRetentionActive, with promtool cases for firing and quiet.

Why 12 GiB and not 4

The first draft used 4 GiB, reasoning from the 1021 MiB currently on disk. That
was wrong: prometheus_tsdb_time_retentions_total is 0, so nothing has ever
been evicted and the store is still filling. Per-block write rates over the
retained window:

block start	MiB/day	series
08-03 → 08-14	16.1–16.9	~3,200–3,500
08-17	37.3	17,382
08-28	72.9	24,005

The step at 08-17 is #62 fixing cAdvisor. Steady state at the current rate is
30 × 72.9 + ~150 MiB WAL/head = 2.28 GiB, or 2.73 GiB once Saruman
gets an agent (#88). 4 GiB would have been 1.5x against an already-planned
estate — a second retention policy wearing a ceiling's clothes, in a repo that
promises 30 days in four places. 12 GiB is 4.4x that and 12% of the volume.

The comment is also explicit about what the flag cannot do: the budget counts
the WAL and m-mapped head chunks but only ever evicts persistent blocks, so a
flood through the remote-write receiver lands somewhere the ceiling can never
reclaim. It bounds sustained growth. Bounding the flood is #182's job.

Verified on the lab, in two deploy cycles

check	result
init / pids_limit applied	init=true on all seven; pids.max read back from the kernel as 512, 1024 on alloy
leak dead	grafana 2495 → 20, zero defunct children, flat across five healthcheck intervals
ceiling armed	prometheus_tsdb_retention_limit_bytes 0 → 12884901888
not truncating	prometheus_tsdb_size_retentions_total = 0
rules	39 loaded, none unhealthy, new rule inactive
no regressions	7/7 up, Grafana HTTP 200, no OOM kills, no restarts

make validate passes. (editorconfig-checker fails locally on four gitignored
alertmanager/.rendered/* files that make render writes without a trailing
newline — pre-existing, untouched here, and invisible to CI, which never
renders. Filed separately.)

Deliberately not done

mem_limit/memswap_limit → #114. cpus prevents nothing at 0.4 of 4 cores.
read_only/cap_drop/security_opt are the group #71 itself calls "worth
deciding on separately". Filed instead:

• ContainerCpuThrottled cannot fire — no service sets a CPU quota #185 — ContainerCpuThrottled is unfireable: it divides CFS counters that
  are permanently zero with no CPU quota set, and has no test. ContainerHighMemory cannot fire — no service sets a memory limit #63 again, one
  rule down in the same file. The fix is a test plus an honest comment, not
  adding cpus to manufacture the condition.
• Run the stack read-only where it can be, and record where it cannot #186 — read_only + tmpfs: five services safe from config inspection,
  grafana needs a soak, alloy pointless while privileged.
• Drop capabilities and set no-new-privileges on the six non-Alloy services #187 — cap_drop: [ALL] + no-new-privileges on the six non-Alloy
  services.
• Reduce Alloy's privilege — it is root on the host and can read the age key #188 — reduce Alloy's privilege. privileged: true + docker socket +
  /:/rootfs is root on the host and read access to the age key. The largest
  item in the file; No resource limits on any container, and no size bound on retention #71 raises it in one clause, and Run the stack read-only where it can be, and record where it cannot #186/Drop capabilities and set no-new-privileges on the six non-Alloy services #187 are worth much
  less until it is done.
• Loki is now the only unbounded store on the disk #189 — Loki has no size bound and is now the only unbounded store on the
  disk.

#114 has a comment recording the two things this settles: its 14-day gate cannot
be satisfied on its own terms, and memswap_limit is decidable today with no
history.

🤖 Generated with Claude Code