fix(observability): take every capability off Alloy (#188)

SECURITY.md

@@ -86,6 +86,13 @@ So `~/.config/sops/age/keys.txt` and the rendered artefacts under
 `snmp-exporter/.rendered/`, `alertmanager/.rendered/` and
 `stacks/observability/.env` — which hold plaintext by design — are all mode 600
 and owned by `robo`, and file permissions are the only thing protecting them.
+
+Which makes anything that can ignore file permissions worth naming. Until
+2026-08-31 the Alloy container was one: uid 0, every capability, and `/` mounted
+read-only, so it could read the age key outright. It now holds no capabilities
+and cannot ([#188](https://github.com/Gerrrt/HomeLab/issues/188)). The Docker
+socket it still mounts is the remaining path — that API can start a container
+with `/` mounted read-write — and closing it is tracked, not done.
 Permissions mean nothing to someone holding the disk.
 
 This is accepted rather than tracked as work. The threat model in

docs/security.md

@@ -242,7 +242,15 @@ complete state table and interface topology. They are credentials.
   published for `oracle`'s agent and remain an accepted residual. See
   [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md).
 - The Alloy debug UI binds to `127.0.0.1` only.
-- The Docker socket is mounted read-only into Alloy.
+- The Docker socket is mounted into Alloy. It is marked `:ro`, which is worth
+  less than it looks: read-only applies to the socket *file*, not to the API
+  behind it, and anything that can talk to that API can start a container with
+  the host filesystem mounted read-write. This is the one remaining path from a
+  compromised Alloy to root on the host — see the paragraph below the list.
+- Alloy holds no capabilities. It runs as uid 0 with `cap_drop: [ALL]` and
+  `no-new-privileges`, so root inside it is subject to file permissions like any
+  other user, and joins only the group that owns `/var/log/syslog` so the auth
+  and syslog sources stay readable (#188).
 - Every service runs under a real init (`init: true`) and a chosen task ceiling
   (`pids_limit`, 512; 1024 for Alloy) rather than the inherited systemd default
   of 9056. This was not theoretical: Grafana's https healthcheck was leaking two
@@ -255,8 +263,22 @@ complete state table and interface topology. They are credentials.
   them.
 - Grafana telemetry and update checks disabled.
 
-Alloy still runs `privileged: true`, which it needs for host-level metric
-collection. That is a real tradeoff and is noted rather than hidden.
+Alloy no longer runs `privileged: true`. It never needed it: `cgroup: host` is
+what makes cAdvisor see the host's cgroups, and dropping every capability
+changed no container metric and cost six unused series —
+`node_rapl_*_joules_total` and `node_cpu_{core,package}_throttles_total`, which
+read root-only sysfs and which nothing here references. Before that change Alloy
+ran as uid 0 with the full capability set, a read-only mount of `/`, and could
+therefore read `~/.config/sops/age/keys.txt` directly. It can no longer.
+
+What remains is the Docker socket, and it is the larger half. Read access to
+that API is enough to create a container with `/` mounted read-write, which is
+root on the host and the age key with it — so the paragraph in `SECURITY.md`
+saying file permissions are all that protect the plaintext artefacts is true of
+every process on the host *except* a compromised Alloy. Putting the socket
+behind a proxy that permits only the handful of GETs cAdvisor and the log
+discovery actually use is tracked separately; the privilege reduction above is
+defence in depth, not a closed door.
 
 ## What this repository deliberately does not publish
 

scripts/render-config.sh

@@ -273,6 +273,20 @@ info "writing $(basename "${STACK_DIR}")/.env"
   # the compose file stay identical on every monitored host, which is the
   # property ADR-0003 leans on.
   printf 'ALLOY_HOSTNAME=%s\n' "$(hostname)"
+
+  # Alloy holds no capabilities since #188, so root inside it is subject to file
+  # permissions like anyone else — and /var/log/auth.log and /var/log/syslog are
+  # syslog:adm 0640, owned by neither. Joining the group that owns them is what
+  # keeps the auth and syslog file sources in config.alloy readable; without it
+  # they fail and most of security.rules.yaml goes quiet, which is the kind of
+  # silence #62 and #63 were both about.
+  #
+  # Derived from the file rather than hardcoded to adm's 4, because compose.yaml
+  # and config.alloy are deployed identically to every monitored host and the
+  # owning group is a property of the host's distribution, not of this
+  # repository. Falls back to 4 only if the file is absent, which on a host with
+  # no syslog is the case where the source has nothing to read anyway.
+  printf 'LOG_READ_GID=%s\n' "$(stat -c '%g' /var/log/syslog 2>/dev/null || echo 4)"
   for var in "${COMPOSE_VARS[@]}"; do
     [[ -n "${!var:-}" ]] && printf '%s=%s\n' "${var}" "${!var}"
   done

scripts/seed-validation-env.sh

@@ -33,6 +33,8 @@ cat "${STACK}/.env.example" > "${OUT}"
   # user, so they are host-specific and deliberately absent from .env.example.
   echo "RENDER_UID=65534"
   echo "RENDER_GID=65534"
+  # Likewise host-specific: render-config.sh reads it off /var/log/syslog.
+  echo "LOG_READ_GID=4"
 } >> "${OUT}"
 
 # Sharing this script stops the two callers drifting from each other. This check

stacks/observability/compose.yaml

@@ -400,14 +400,67 @@ services:
     # cgroups, tails the journal and talks to the Docker socket, so it has more
     # reason than the others to fan out. Observed peak is 20 threads either way.
     pids_limit: 1024
-    privileged: true
+    # `privileged: true` used to be here, and it was never what made cAdvisor
+    # work — cgroup: host below is. What it did do was hand every capability to
+    # the one container holding the Docker socket and a read-only mount of the
+    # whole host filesystem. Measured on 2026-08-31: uid 0, CapEff
+    # 000001ffffffffff, and /rootfs/home/robo/.config/sops/age/keys.txt
+    # readable — the key every secret in this repository decrypts with, which
+    # SECURITY.md says file permissions alone protect. Dropping the capability
+    # set closes that read, because root without DAC_OVERRIDE is subject to the
+    # 0600 like anyone else (#188).
+    #
+    # Cost, measured rather than assumed: six series. node_rapl_*_joules_total
+    # (3) and node_cpu_{core,package}_throttles_total (3) read sysfs files that
+    # are root-only, and node_exporter drops them silently — no collector
+    # failure, no log line. Nothing in this repository references any of the
+    # six. Everything else is identical: 895 cAdvisor series, seven container
+    # names, the same node_filesystem mountpoints.
+    cap_drop: [ALL]
+    security_opt:
+      - no-new-privileges:true
+    # Root without DAC_OVERRIDE also cannot read /var/log/auth.log or
+    # /var/log/syslog — they are syslog:adm 0640 and root owns neither — which
+    # would take out the two file sources in config.alloy and, with them, most
+    # of security.rules.yaml. Joining the group that owns them restores exactly
+    # that and nothing else.
+    #
+    # The tempting alternative is cap_add: [DAC_READ_SEARCH]. Do not: it grants
+    # read of *every* file on the mounted rootfs, which puts the age key back
+    # within reach and undoes the whole point. Both were tested; the group
+    # leaves the key denied and the capability does not.
+    #
+    # Rendered rather than hardcoded to 4 because config.alloy and this file are
+    # deployed identically to every monitored host, and the group that owns the
+    # syslog files is a property of the host, not of this repository.
+    group_add:
+      - "${LOG_READ_GID:?run make render}"
+      # 473 is the image's own `alloy` group, and alloy-data is 0770 473:473 —
+      # so without this the agent cannot write its own WAL and positions files
+      # and dies on `mkdir /var/lib/alloy/data: permission denied` before it
+      # reads a line of config. Under `privileged: true` DAC_OVERRIDE hid that,
+      # which is a fair summary of what that setting was doing here generally.
+      #
+      # A property of the pinned image rather than of the host, hence the
+      # literal. Re-derive after an image bump with:
+      #   docker run --rm --entrypoint sh <image> -c 'stat -c %g /var/lib/alloy'
+      # Getting it wrong is at least loud: the container crash-loops from the
+      # first second and ContainerRestartLoop fires, rather than coming up and
+      # collecting nothing.
+      - "473"
     # Without this the cAdvisor exporter reports exactly one series, id="/",
     # and every per-container metric is missing — which is what the Docker
-    # Containers dashboard and four of the eight rules in containers.rules.yaml
-    # are built on. `privileged: true` is not enough: on cgroup v2 Docker's
-    # default namespace mode is `private`, so the container's own cgroup IS the
-    # root it can see, and cAdvisor faithfully reports the one cgroup it finds.
-    # Nothing errors. The exporter stays healthy and the dashboard stays empty.
+    # Containers dashboard and four of the nine rules in containers.rules.yaml
+    # are built on. On cgroup v2 Docker's default namespace mode is `private`,
+    # so the container's own cgroup IS the root it can see, and cAdvisor
+    # faithfully reports the one cgroup it finds. Nothing errors. The exporter
+    # stays healthy and the dashboard stays empty.
+    #
+    # This is the setting that matters, and it is not a privilege: the
+    # `privileged: true` that sat above was neither necessary nor sufficient for
+    # it. Sufficiency is what #62 found — privileged alone still gave one
+    # series. Necessity is what #188 measured — dropping every capability
+    # changed no container metric at all.
     cgroup: host
     environment:
       LOKI_URL: http://loki:3100/loki/api/v1/push
@@ -425,16 +478,32 @@ services:
     volumes:
       - ./alloy/config.alloy:/etc/alloy/config.alloy:ro
       - alloy-data:/var/lib/alloy/data
+      # The API, not the log files: cAdvisor takes container names and labels
+      # from it, and discovery.docker + loki.source.docker stream stdout through
+      # it. `:ro` is close to decorative — it applies to the socket file, not to
+      # the API behind it, and that API can create a container with / mounted
+      # read-write. It is the one path left from a compromised Alloy to root on
+      # this host, and it is why the capability drop above is defence in depth
+      # rather than a closed door. Tracked separately; see docs/security.md.
       - /var/run/docker.sock:/var/run/docker.sock:ro
-      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      # /var/lib/docker/containers is NOT mounted, though it was until #188.
+      # Nothing read it: loki.source.docker streams over the socket above rather
+      # than tailing the json-file driver's output, config.alloy names the path
+      # only to *exclude* it from the filesystem collector, and cAdvisor was
+      # measured producing identical series without it. It is root-only and
+      # holds every container's logs, so an unread mount of it is pure reach.
       - /var/log:/var/log:ro
       - /:/rootfs:ro
     ports:
       - "127.0.0.1:${ALLOY_PORT:-12345}:12345"
       # Network syslog for devices that cannot run an agent — morpheus today.
       # Unlike the debug UI above this must be reachable off-host, so it binds
-      # to BIND_ADDR. 1514 rather than 514: the container does not run as root
-      # and cannot take a privileged port. pfSense sends UDP by default.
+      # to BIND_ADDR. 1514 rather than 514: the container cannot take a
+      # privileged port. That claim used to read "does not run as root", which
+      # was simply false — it runs as uid 0 and, until #188, held every
+      # capability including NET_BIND_SERVICE. It is true now, and true for the
+      # stated reason, because cap_drop above removes that capability.
+      # pfSense sends UDP by default.
       - "${BIND_ADDR:-0.0.0.0}:${SYSLOG_PORT:-1514}:1514/udp"
     depends_on:
       # service_started for loki, for the same reason as grafana above: the