From 7d5a94376c52e2b2e41428fc2334408385515b94 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 31 Aug 2026 02:22:30 +0000 Subject: [PATCH] fix(observability): take every capability off Alloy (#188) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Alloy held `privileged: true`, the Docker socket, and a read-only mount of the whole host filesystem. Measured before changing anything: uid 0, CapEff 000001ffffffffff, and ~/.config/sops/age/keys.txt readable straight off /rootfs — the key every secret in this repository decrypts with, and the one SECURITY.md says file permissions alone protect. They did not protect it from this container. `privileged: true` was never what made cAdvisor work. `cgroup: host` is, and that is not a privilege. #62 already proved privileged was not sufficient — it gave one series, id="/", until the cgroup namespace was fixed. This proves it was not necessary either: with cap_drop ALL the exporter returns the same 895 series, the same seven container names, the same eleven ids, the same node_filesystem mountpoints. The key is then denied, because root without DAC_OVERRIDE is subject to a 0600 like anyone else. The cost is six series, measured rather than assumed: node_rapl_*_joules_total and node_cpu_{core,package}_throttles_total read root-only sysfs, and node_exporter drops them silently — no collector failure, no log line. Nothing in this repository references any of the six. That silence is exactly why the number is written into compose.yaml rather than left to be discovered. Two things had to be given back, and finding them is the whole risk of this change: - /var/log/auth.log and /var/log/syslog are syslog:adm 0640 and root owns neither, so dropping DAC_OVERRIDE takes out both file sources and most of security.rules.yaml with them. group_add on the owning group restores exactly those. cap_add: [DAC_READ_SEARCH] would too, and would hand back the age key — tested both ways, which is the only reason to trust the choice. - alloy-data is 0770 473:473, so the agent could not create its own WAL directory and crash-looped on `mkdir /var/lib/alloy/data: permission denied` before reading a line of config. That is what privileged had been hiding. Joining the image's own group fixes it. The log gid is rendered from /var/log/syslog rather than hardcoded to 4, because compose.yaml and config.alloy deploy identically to every monitored host and the owning group is a property of the distribution. /var/lib/docker/containers is gone. Nothing read it: loki.source.docker streams over the socket, config.alloy names the path only to exclude it from the filesystem collector, and cAdvisor was measured producing identical series without it. It is root-only and holds every container's logs, so an unread mount of it was pure reach. Also corrects a comment that asserted a security property the file did not have: the syslog port note claimed the container "does not run as root". It ran as uid 0 with every capability. The claim is true now, and for the reason it now gives. What this does not close is the Docker socket, and the docs now say so instead of implying otherwise. `:ro` there applies to the socket file, not the API, and that API can create a container with / mounted read-write — root on the host and the age key with it. Putting it behind an allowlisting proxy is tracked separately; this is defence in depth, not a closed door. Verified on the host against a baseline captured before any change: seven container names before and after, node series 669 -> 663 (the six above), auth.log and syslog markers written with `logger` and found in Loki, journal 97 lines/hr against 82 before, docker logs 688 lines/3m, 13 targets up, 39 rules healthy, no restarts. Co-Authored-By: Claude Opus 5 --- SECURITY.md | 7 +++ docs/security.md | 28 ++++++++-- scripts/render-config.sh | 14 +++++ scripts/seed-validation-env.sh | 2 + stacks/observability/compose.yaml | 87 +++++++++++++++++++++++++++---- 5 files changed, 126 insertions(+), 12 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 94def5b..62d25cc 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -86,6 +86,13 @@ So `~/.config/sops/age/keys.txt` and the rendered artefacts under `snmp-exporter/.rendered/`, `alertmanager/.rendered/` and `stacks/observability/.env` — which hold plaintext by design — are all mode 600 and owned by `robo`, and file permissions are the only thing protecting them. + +Which makes anything that can ignore file permissions worth naming. Until +2026-08-31 the Alloy container was one: uid 0, every capability, and `/` mounted +read-only, so it could read the age key outright. It now holds no capabilities +and cannot ([#188](https://github.com/Gerrrt/HomeLab/issues/188)). The Docker +socket it still mounts is the remaining path — that API can start a container +with `/` mounted read-write — and closing it is tracked, not done. Permissions mean nothing to someone holding the disk. This is accepted rather than tracked as work. The threat model in diff --git a/docs/security.md b/docs/security.md index 5b4b803..714a71b 100644 --- a/docs/security.md +++ b/docs/security.md @@ -242,7 +242,15 @@ complete state table and interface topology. They are credentials. published for `oracle`'s agent and remain an accepted residual. See [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). - The Alloy debug UI binds to `127.0.0.1` only. -- The Docker socket is mounted read-only into Alloy. +- The Docker socket is mounted into Alloy. It is marked `:ro`, which is worth + less than it looks: read-only applies to the socket *file*, not to the API + behind it, and anything that can talk to that API can start a container with + the host filesystem mounted read-write. This is the one remaining path from a + compromised Alloy to root on the host — see the paragraph below the list. +- Alloy holds no capabilities. It runs as uid 0 with `cap_drop: [ALL]` and + `no-new-privileges`, so root inside it is subject to file permissions like any + other user, and joins only the group that owns `/var/log/syslog` so the auth + and syslog sources stay readable (#188). - Every service runs under a real init (`init: true`) and a chosen task ceiling (`pids_limit`, 512; 1024 for Alloy) rather than the inherited systemd default of 9056. This was not theoretical: Grafana's https healthcheck was leaking two @@ -255,8 +263,22 @@ complete state table and interface topology. They are credentials. them. - Grafana telemetry and update checks disabled. -Alloy still runs `privileged: true`, which it needs for host-level metric -collection. That is a real tradeoff and is noted rather than hidden. +Alloy no longer runs `privileged: true`. It never needed it: `cgroup: host` is +what makes cAdvisor see the host's cgroups, and dropping every capability +changed no container metric and cost six unused series — +`node_rapl_*_joules_total` and `node_cpu_{core,package}_throttles_total`, which +read root-only sysfs and which nothing here references. Before that change Alloy +ran as uid 0 with the full capability set, a read-only mount of `/`, and could +therefore read `~/.config/sops/age/keys.txt` directly. It can no longer. + +What remains is the Docker socket, and it is the larger half. Read access to +that API is enough to create a container with `/` mounted read-write, which is +root on the host and the age key with it — so the paragraph in `SECURITY.md` +saying file permissions are all that protect the plaintext artefacts is true of +every process on the host *except* a compromised Alloy. Putting the socket +behind a proxy that permits only the handful of GETs cAdvisor and the log +discovery actually use is tracked separately; the privilege reduction above is +defence in depth, not a closed door. ## What this repository deliberately does not publish diff --git a/scripts/render-config.sh b/scripts/render-config.sh index b789150..c0e7e4e 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -273,6 +273,20 @@ info "writing $(basename "${STACK_DIR}")/.env" # the compose file stay identical on every monitored host, which is the # property ADR-0003 leans on. printf 'ALLOY_HOSTNAME=%s\n' "$(hostname)" + + # Alloy holds no capabilities since #188, so root inside it is subject to file + # permissions like anyone else — and /var/log/auth.log and /var/log/syslog are + # syslog:adm 0640, owned by neither. Joining the group that owns them is what + # keeps the auth and syslog file sources in config.alloy readable; without it + # they fail and most of security.rules.yaml goes quiet, which is the kind of + # silence #62 and #63 were both about. + # + # Derived from the file rather than hardcoded to adm's 4, because compose.yaml + # and config.alloy are deployed identically to every monitored host and the + # owning group is a property of the host's distribution, not of this + # repository. Falls back to 4 only if the file is absent, which on a host with + # no syslog is the case where the source has nothing to read anyway. + printf 'LOG_READ_GID=%s\n' "$(stat -c '%g' /var/log/syslog 2>/dev/null || echo 4)" for var in "${COMPOSE_VARS[@]}"; do [[ -n "${!var:-}" ]] && printf '%s=%s\n' "${var}" "${!var}" done diff --git a/scripts/seed-validation-env.sh b/scripts/seed-validation-env.sh index 35eca81..e420a1d 100755 --- a/scripts/seed-validation-env.sh +++ b/scripts/seed-validation-env.sh @@ -33,6 +33,8 @@ cat "${STACK}/.env.example" > "${OUT}" # user, so they are host-specific and deliberately absent from .env.example. echo "RENDER_UID=65534" echo "RENDER_GID=65534" + # Likewise host-specific: render-config.sh reads it off /var/log/syslog. + echo "LOG_READ_GID=4" } >> "${OUT}" # Sharing this script stops the two callers drifting from each other. This check diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 0710df9..5dce859 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -400,14 +400,67 @@ services: # cgroups, tails the journal and talks to the Docker socket, so it has more # reason than the others to fan out. Observed peak is 20 threads either way. pids_limit: 1024 - privileged: true + # `privileged: true` used to be here, and it was never what made cAdvisor + # work — cgroup: host below is. What it did do was hand every capability to + # the one container holding the Docker socket and a read-only mount of the + # whole host filesystem. Measured on 2026-08-31: uid 0, CapEff + # 000001ffffffffff, and /rootfs/home/robo/.config/sops/age/keys.txt + # readable — the key every secret in this repository decrypts with, which + # SECURITY.md says file permissions alone protect. Dropping the capability + # set closes that read, because root without DAC_OVERRIDE is subject to the + # 0600 like anyone else (#188). + # + # Cost, measured rather than assumed: six series. node_rapl_*_joules_total + # (3) and node_cpu_{core,package}_throttles_total (3) read sysfs files that + # are root-only, and node_exporter drops them silently — no collector + # failure, no log line. Nothing in this repository references any of the + # six. Everything else is identical: 895 cAdvisor series, seven container + # names, the same node_filesystem mountpoints. + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + # Root without DAC_OVERRIDE also cannot read /var/log/auth.log or + # /var/log/syslog — they are syslog:adm 0640 and root owns neither — which + # would take out the two file sources in config.alloy and, with them, most + # of security.rules.yaml. Joining the group that owns them restores exactly + # that and nothing else. + # + # The tempting alternative is cap_add: [DAC_READ_SEARCH]. Do not: it grants + # read of *every* file on the mounted rootfs, which puts the age key back + # within reach and undoes the whole point. Both were tested; the group + # leaves the key denied and the capability does not. + # + # Rendered rather than hardcoded to 4 because config.alloy and this file are + # deployed identically to every monitored host, and the group that owns the + # syslog files is a property of the host, not of this repository. + group_add: + - "${LOG_READ_GID:?run make render}" + # 473 is the image's own `alloy` group, and alloy-data is 0770 473:473 — + # so without this the agent cannot write its own WAL and positions files + # and dies on `mkdir /var/lib/alloy/data: permission denied` before it + # reads a line of config. Under `privileged: true` DAC_OVERRIDE hid that, + # which is a fair summary of what that setting was doing here generally. + # + # A property of the pinned image rather than of the host, hence the + # literal. Re-derive after an image bump with: + # docker run --rm --entrypoint sh -c 'stat -c %g /var/lib/alloy' + # Getting it wrong is at least loud: the container crash-loops from the + # first second and ContainerRestartLoop fires, rather than coming up and + # collecting nothing. + - "473" # Without this the cAdvisor exporter reports exactly one series, id="/", # and every per-container metric is missing — which is what the Docker - # Containers dashboard and four of the eight rules in containers.rules.yaml - # are built on. `privileged: true` is not enough: on cgroup v2 Docker's - # default namespace mode is `private`, so the container's own cgroup IS the - # root it can see, and cAdvisor faithfully reports the one cgroup it finds. - # Nothing errors. The exporter stays healthy and the dashboard stays empty. + # Containers dashboard and four of the nine rules in containers.rules.yaml + # are built on. On cgroup v2 Docker's default namespace mode is `private`, + # so the container's own cgroup IS the root it can see, and cAdvisor + # faithfully reports the one cgroup it finds. Nothing errors. The exporter + # stays healthy and the dashboard stays empty. + # + # This is the setting that matters, and it is not a privilege: the + # `privileged: true` that sat above was neither necessary nor sufficient for + # it. Sufficiency is what #62 found — privileged alone still gave one + # series. Necessity is what #188 measured — dropping every capability + # changed no container metric at all. cgroup: host environment: LOKI_URL: http://loki:3100/loki/api/v1/push @@ -425,16 +478,32 @@ services: volumes: - ./alloy/config.alloy:/etc/alloy/config.alloy:ro - alloy-data:/var/lib/alloy/data + # The API, not the log files: cAdvisor takes container names and labels + # from it, and discovery.docker + loki.source.docker stream stdout through + # it. `:ro` is close to decorative — it applies to the socket file, not to + # the API behind it, and that API can create a container with / mounted + # read-write. It is the one path left from a compromised Alloy to root on + # this host, and it is why the capability drop above is defence in depth + # rather than a closed door. Tracked separately; see docs/security.md. - /var/run/docker.sock:/var/run/docker.sock:ro - - /var/lib/docker/containers:/var/lib/docker/containers:ro + # /var/lib/docker/containers is NOT mounted, though it was until #188. + # Nothing read it: loki.source.docker streams over the socket above rather + # than tailing the json-file driver's output, config.alloy names the path + # only to *exclude* it from the filesystem collector, and cAdvisor was + # measured producing identical series without it. It is root-only and + # holds every container's logs, so an unread mount of it is pure reach. - /var/log:/var/log:ro - /:/rootfs:ro ports: - "127.0.0.1:${ALLOY_PORT:-12345}:12345" # Network syslog for devices that cannot run an agent — morpheus today. # Unlike the debug UI above this must be reachable off-host, so it binds - # to BIND_ADDR. 1514 rather than 514: the container does not run as root - # and cannot take a privileged port. pfSense sends UDP by default. + # to BIND_ADDR. 1514 rather than 514: the container cannot take a + # privileged port. That claim used to read "does not run as root", which + # was simply false — it runs as uid 0 and, until #188, held every + # capability including NET_BIND_SERVICE. It is true now, and true for the + # stated reason, because cap_drop above removes that capability. + # pfSense sends UDP by default. - "${BIND_ADDR:-0.0.0.0}:${SYSLOG_PORT:-1514}:1514/udp" depends_on: # service_started for loki, for the same reason as grafana above: the