diff --git a/SECURITY.md b/SECURITY.md index 94def5b..62d25cc 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -86,6 +86,13 @@ So `~/.config/sops/age/keys.txt` and the rendered artefacts under `snmp-exporter/.rendered/`, `alertmanager/.rendered/` and `stacks/observability/.env` — which hold plaintext by design — are all mode 600 and owned by `robo`, and file permissions are the only thing protecting them. + +Which makes anything that can ignore file permissions worth naming. Until +2026-08-31 the Alloy container was one: uid 0, every capability, and `/` mounted +read-only, so it could read the age key outright. It now holds no capabilities +and cannot ([#188](https://github.com/Gerrrt/HomeLab/issues/188)). The Docker +socket it still mounts is the remaining path — that API can start a container +with `/` mounted read-write — and closing it is tracked, not done. Permissions mean nothing to someone holding the disk. This is accepted rather than tracked as work. The threat model in diff --git a/docs/security.md b/docs/security.md index 5b4b803..714a71b 100644 --- a/docs/security.md +++ b/docs/security.md @@ -242,7 +242,15 @@ complete state table and interface topology. They are credentials. published for `oracle`'s agent and remain an accepted residual. See [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). - The Alloy debug UI binds to `127.0.0.1` only. -- The Docker socket is mounted read-only into Alloy. +- The Docker socket is mounted into Alloy. It is marked `:ro`, which is worth + less than it looks: read-only applies to the socket *file*, not to the API + behind it, and anything that can talk to that API can start a container with + the host filesystem mounted read-write. This is the one remaining path from a + compromised Alloy to root on the host — see the paragraph below the list. +- Alloy holds no capabilities. It runs as uid 0 with `cap_drop: [ALL]` and + `no-new-privileges`, so root inside it is subject to file permissions like any + other user, and joins only the group that owns `/var/log/syslog` so the auth + and syslog sources stay readable (#188). - Every service runs under a real init (`init: true`) and a chosen task ceiling (`pids_limit`, 512; 1024 for Alloy) rather than the inherited systemd default of 9056. This was not theoretical: Grafana's https healthcheck was leaking two @@ -255,8 +263,22 @@ complete state table and interface topology. They are credentials. them. - Grafana telemetry and update checks disabled. -Alloy still runs `privileged: true`, which it needs for host-level metric -collection. That is a real tradeoff and is noted rather than hidden. +Alloy no longer runs `privileged: true`. It never needed it: `cgroup: host` is +what makes cAdvisor see the host's cgroups, and dropping every capability +changed no container metric and cost six unused series — +`node_rapl_*_joules_total` and `node_cpu_{core,package}_throttles_total`, which +read root-only sysfs and which nothing here references. Before that change Alloy +ran as uid 0 with the full capability set, a read-only mount of `/`, and could +therefore read `~/.config/sops/age/keys.txt` directly. It can no longer. + +What remains is the Docker socket, and it is the larger half. Read access to +that API is enough to create a container with `/` mounted read-write, which is +root on the host and the age key with it — so the paragraph in `SECURITY.md` +saying file permissions are all that protect the plaintext artefacts is true of +every process on the host *except* a compromised Alloy. Putting the socket +behind a proxy that permits only the handful of GETs cAdvisor and the log +discovery actually use is tracked separately; the privilege reduction above is +defence in depth, not a closed door. ## What this repository deliberately does not publish diff --git a/scripts/render-config.sh b/scripts/render-config.sh index b789150..c0e7e4e 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -273,6 +273,20 @@ info "writing $(basename "${STACK_DIR}")/.env" # the compose file stay identical on every monitored host, which is the # property ADR-0003 leans on. printf 'ALLOY_HOSTNAME=%s\n' "$(hostname)" + + # Alloy holds no capabilities since #188, so root inside it is subject to file + # permissions like anyone else — and /var/log/auth.log and /var/log/syslog are + # syslog:adm 0640, owned by neither. Joining the group that owns them is what + # keeps the auth and syslog file sources in config.alloy readable; without it + # they fail and most of security.rules.yaml goes quiet, which is the kind of + # silence #62 and #63 were both about. + # + # Derived from the file rather than hardcoded to adm's 4, because compose.yaml + # and config.alloy are deployed identically to every monitored host and the + # owning group is a property of the host's distribution, not of this + # repository. Falls back to 4 only if the file is absent, which on a host with + # no syslog is the case where the source has nothing to read anyway. + printf 'LOG_READ_GID=%s\n' "$(stat -c '%g' /var/log/syslog 2>/dev/null || echo 4)" for var in "${COMPOSE_VARS[@]}"; do [[ -n "${!var:-}" ]] && printf '%s=%s\n' "${var}" "${!var}" done diff --git a/scripts/seed-validation-env.sh b/scripts/seed-validation-env.sh index 35eca81..e420a1d 100755 --- a/scripts/seed-validation-env.sh +++ b/scripts/seed-validation-env.sh @@ -33,6 +33,8 @@ cat "${STACK}/.env.example" > "${OUT}" # user, so they are host-specific and deliberately absent from .env.example. echo "RENDER_UID=65534" echo "RENDER_GID=65534" + # Likewise host-specific: render-config.sh reads it off /var/log/syslog. + echo "LOG_READ_GID=4" } >> "${OUT}" # Sharing this script stops the two callers drifting from each other. This check diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 0710df9..5dce859 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -400,14 +400,67 @@ services: # cgroups, tails the journal and talks to the Docker socket, so it has more # reason than the others to fan out. Observed peak is 20 threads either way. pids_limit: 1024 - privileged: true + # `privileged: true` used to be here, and it was never what made cAdvisor + # work — cgroup: host below is. What it did do was hand every capability to + # the one container holding the Docker socket and a read-only mount of the + # whole host filesystem. Measured on 2026-08-31: uid 0, CapEff + # 000001ffffffffff, and /rootfs/home/robo/.config/sops/age/keys.txt + # readable — the key every secret in this repository decrypts with, which + # SECURITY.md says file permissions alone protect. Dropping the capability + # set closes that read, because root without DAC_OVERRIDE is subject to the + # 0600 like anyone else (#188). + # + # Cost, measured rather than assumed: six series. node_rapl_*_joules_total + # (3) and node_cpu_{core,package}_throttles_total (3) read sysfs files that + # are root-only, and node_exporter drops them silently — no collector + # failure, no log line. Nothing in this repository references any of the + # six. Everything else is identical: 895 cAdvisor series, seven container + # names, the same node_filesystem mountpoints. + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + # Root without DAC_OVERRIDE also cannot read /var/log/auth.log or + # /var/log/syslog — they are syslog:adm 0640 and root owns neither — which + # would take out the two file sources in config.alloy and, with them, most + # of security.rules.yaml. Joining the group that owns them restores exactly + # that and nothing else. + # + # The tempting alternative is cap_add: [DAC_READ_SEARCH]. Do not: it grants + # read of *every* file on the mounted rootfs, which puts the age key back + # within reach and undoes the whole point. Both were tested; the group + # leaves the key denied and the capability does not. + # + # Rendered rather than hardcoded to 4 because config.alloy and this file are + # deployed identically to every monitored host, and the group that owns the + # syslog files is a property of the host, not of this repository. + group_add: + - "${LOG_READ_GID:?run make render}" + # 473 is the image's own `alloy` group, and alloy-data is 0770 473:473 — + # so without this the agent cannot write its own WAL and positions files + # and dies on `mkdir /var/lib/alloy/data: permission denied` before it + # reads a line of config. Under `privileged: true` DAC_OVERRIDE hid that, + # which is a fair summary of what that setting was doing here generally. + # + # A property of the pinned image rather than of the host, hence the + # literal. Re-derive after an image bump with: + # docker run --rm --entrypoint sh -c 'stat -c %g /var/lib/alloy' + # Getting it wrong is at least loud: the container crash-loops from the + # first second and ContainerRestartLoop fires, rather than coming up and + # collecting nothing. + - "473" # Without this the cAdvisor exporter reports exactly one series, id="/", # and every per-container metric is missing — which is what the Docker - # Containers dashboard and four of the eight rules in containers.rules.yaml - # are built on. `privileged: true` is not enough: on cgroup v2 Docker's - # default namespace mode is `private`, so the container's own cgroup IS the - # root it can see, and cAdvisor faithfully reports the one cgroup it finds. - # Nothing errors. The exporter stays healthy and the dashboard stays empty. + # Containers dashboard and four of the nine rules in containers.rules.yaml + # are built on. On cgroup v2 Docker's default namespace mode is `private`, + # so the container's own cgroup IS the root it can see, and cAdvisor + # faithfully reports the one cgroup it finds. Nothing errors. The exporter + # stays healthy and the dashboard stays empty. + # + # This is the setting that matters, and it is not a privilege: the + # `privileged: true` that sat above was neither necessary nor sufficient for + # it. Sufficiency is what #62 found — privileged alone still gave one + # series. Necessity is what #188 measured — dropping every capability + # changed no container metric at all. cgroup: host environment: LOKI_URL: http://loki:3100/loki/api/v1/push @@ -425,16 +478,32 @@ services: volumes: - ./alloy/config.alloy:/etc/alloy/config.alloy:ro - alloy-data:/var/lib/alloy/data + # The API, not the log files: cAdvisor takes container names and labels + # from it, and discovery.docker + loki.source.docker stream stdout through + # it. `:ro` is close to decorative — it applies to the socket file, not to + # the API behind it, and that API can create a container with / mounted + # read-write. It is the one path left from a compromised Alloy to root on + # this host, and it is why the capability drop above is defence in depth + # rather than a closed door. Tracked separately; see docs/security.md. - /var/run/docker.sock:/var/run/docker.sock:ro - - /var/lib/docker/containers:/var/lib/docker/containers:ro + # /var/lib/docker/containers is NOT mounted, though it was until #188. + # Nothing read it: loki.source.docker streams over the socket above rather + # than tailing the json-file driver's output, config.alloy names the path + # only to *exclude* it from the filesystem collector, and cAdvisor was + # measured producing identical series without it. It is root-only and + # holds every container's logs, so an unread mount of it is pure reach. - /var/log:/var/log:ro - /:/rootfs:ro ports: - "127.0.0.1:${ALLOY_PORT:-12345}:12345" # Network syslog for devices that cannot run an agent — morpheus today. # Unlike the debug UI above this must be reachable off-host, so it binds - # to BIND_ADDR. 1514 rather than 514: the container does not run as root - # and cannot take a privileged port. pfSense sends UDP by default. + # to BIND_ADDR. 1514 rather than 514: the container cannot take a + # privileged port. That claim used to read "does not run as root", which + # was simply false — it runs as uid 0 and, until #188, held every + # capability including NET_BIND_SERVICE. It is true now, and true for the + # stated reason, because cap_drop above removes that capability. + # pfSense sends UDP by default. - "${BIND_ADDR:-0.0.0.0}:${SYSLOG_PORT:-1514}:1514/udp" depends_on: # service_started for loki, for the same reason as grafana above: the