fix(observability): take every capability off Alloy (#188)

[Gerrrt]

Closes #188.

Alloy held privileged: true, the Docker socket, and a read-only mount of the
whole host filesystem. Measured before changing anything:

uid=0(root)  CapEff: 000001ffffffffff
$ docker exec alloy head -c1 /rootfs/home/robo/.config/sops/age/keys.txt
(readable)

That is the key every secret in this repository decrypts with, and the one
SECURITY.md says file permissions alone protect. They did not protect it from
this container.

privileged: true was never doing the job it was credited with

cgroup: host is what makes cAdvisor see the host's cgroups, and that is not a
privilege. #62 already showed privileged was not sufficient — it gave one
series, id="/", until the cgroup namespace was fixed. This change shows it was
not necessary either.

Tested on a throwaway container first, so the live agent was never the
experiment:

config	container names	ids	node series	age key
--privileged (before)	8	12	658	READABLE
no privileged	8	12	652	READABLE
--cap-drop ALL	8	12	652	denied
+ no-new-privileges	8	12	652	denied
non-root (65534)	0	4	652	denied

Root is kept: dropping to a non-root uid reproduces the #62 symptom exactly.
Dropping capabilities does not.

Cost: six series, and nothing uses them

node_rapl_{core,package,uncore}_joules_total and
node_cpu_{core,package}_throttles_total read root-only sysfs.
node_exporter drops them silently — no collector failure, no log line. Zero
references anywhere in the repo. The number is written into compose.yaml
precisely because the loss is silent.

Two things had to be given back — finding them is the whole risk

• auth.log and syslog are syslog:adm 0640 and root owns neither, so
  dropping DAC_OVERRIDE takes out both file sources and most of
  security.rules.yaml. group_add on the owning group restores exactly those.
  cap_add: [DAC_READ_SEARCH] would restore them too — and hand back the age
  key. Tested both ways; that is the only reason to trust the choice.
• alloy-data is 0770 473:473, so the agent could not create its own WAL
  directory and crash-looped on mkdir /var/lib/alloy/data: permission denied
  before reading a line of config. That is what privileged had been hiding.
  Joining the image's own group fixes it.

The log gid is rendered from /var/log/syslog rather than hardcoded to 4,
because compose.yaml and config.alloy deploy identically to every monitored
host and the owning group is a property of the distribution.

/var/lib/docker/containers is gone

Nothing read it: loki.source.docker streams over the socket, config.alloy
names the path only to exclude it from the filesystem collector, and cAdvisor
was measured producing identical series without it. Root-only, holds every
container's logs, so an unread mount of it was pure reach.

A comment that asserted a property the file did not have

The syslog-port note claimed the container "does not run as root". It ran as uid
0 with every capability including NET_BIND_SERVICE. The claim is true now, and
for the reason it now gives.

What this does not close

The Docker socket. :ro there applies to the socket file, not the API, and
that API can create a container with / mounted read-write — root on the host
and the age key with it. The docs now say this plainly instead of listing the
read-only socket as a mitigation. An allowlisting proxy is #193.
This is defence in depth, not a closed door.

Verified on the host, against a baseline captured before any change

check	before	after
distinct container names	7	7
node series	669	663 (the six above)
node_filesystem mountpoints	/ /boot /boot/efi …	identical
age key	readable	denied, CapEff 0000000000000000
auth.log	flowing	logger marker written and found in Loki
syslog	flowing	logger marker written and found in Loki
journal	82 lines/hr	97 lines/hr
docker logs	flowing	688 lines/3m
targets / rules	13 up / 39 healthy	13 up / 39 healthy

make validate green. container_cpu_cfs_periods_total is absent both before
and after — that is #185, not this.

One thing found while verifying and filed rather than fixed here: Alloy is
delivering ~1.5% of the host journal (#194). Unchanged by this PR — 82 lines/hr
before, 97 after — but it is the #62 shape and worth its own look.

🤖 Generated with Claude Code