From aa20d1379bdbde1ddee5b2c966d09c98fbe08fe2 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 30 Aug 2026 22:56:56 +0000 Subject: [PATCH 1/2] fix(observability): bind the unauthenticated ports to loopback (#70) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prometheus, Alertmanager and Loki published on ${BIND_ADDR}, which shipped as 0.0.0.0. None of the three authenticates: Prometheus carries a writable remote-write receiver, Loki runs auth_enabled: false with a delete API, and Alertmanager's silence API is how monitoring gets switched off — quietly, using the system that would otherwise have recorded the act. Default-deny between VLANs was the only control, and ADR-0002 already records the weakness that leaves: "a compromised workstation reaches Winterfell". One firewall rule stood between an ordinary desktop compromise and write access to the metric and log stores. Nothing off the monitoring host consumed any of the three. Scrape targets, Grafana's datasources and the local Alloy all address services by compose name on the internal network; morpheus arrives by syslog on 1514/udp, not via Loki. The wide bind was buying reach nobody was using. So publish only what has an off-host consumer. Grafana and the syslog receiver keep ${BIND_ADDR}; the other three take a literal 127.0.0.1 beside the Alloy debug UI that was already there. Hard-coded rather than made a variable: render-config.sh regenerates .env from .env.example on every `make up`, so a variable would advertise a knob the deploy path does not provide, and widening one of these hands a whole VLAN write access to the stores — that belongs in a reviewed diff. Silences stay reachable off-host through Grafana, which proxies Alertmanager over the compose network behind a login, so this costs no capability. Co-Authored-By: Claude Opus 5 --- README.md | 7 +- SECURITY.md | 1 + ...d-the-unauthenticated-ports-to-loopback.md | 87 +++++++++++++++++++ docs/architecture.md | 30 +++++-- docs/observability.md | 7 +- docs/runbooks/add-monitored-device.md | 25 ++++-- docs/runbooks/deploy-stack.md | 10 +++ docs/runbooks/restore-the-stack.md | 4 + docs/security.md | 14 +++ stacks/observability/.env.example | 22 +++-- stacks/observability/README.md | 15 +++- stacks/observability/compose.yaml | 39 +++++++-- 12 files changed, 229 insertions(+), 32 deletions(-) create mode 100644 docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md diff --git a/README.md b/README.md index b276415..7532774 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ rack; a dashed border means egress only. Full topology and data flow in ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 11 architecture decision records +│ ├── adr/ # 12 architecture decision records │ └── runbooks/ # deploy, add device, rotate creds, certs, key backup, │ # purge, restore the firewall, restore the stack, │ # ship firewall logs, verify the alert path, @@ -195,7 +195,10 @@ Prometheus verifies it with the CA, so `make up` renders nothing until they exist. Details in [`docs/runbooks/generate-certificates.md`](docs/runbooks/generate-certificates.md). -Grafana on `:3000` over https, Prometheus on `:9090`. Grafana's certificate is +Grafana on `:3000` over https, from any workstation that can route to the host. +Prometheus, Loki and Alertmanager bind to `127.0.0.1` and are reached from the +host itself or through Grafana +([#70](https://github.com/Gerrrt/HomeLab/issues/70)). Grafana's certificate is signed by the lab's own CA, so a browser warns and `curl` needs `-k` until you trust `certificates/ca.pem` — step 4 of that runbook. Full procedure, verification steps and troubleshooting in diff --git a/SECURITY.md b/SECURITY.md index 8ec89b5..11c44fa 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -44,6 +44,7 @@ is a very different thing from an overlooked one. Full detail in | Decrypted secrets in editor undo files, written by `make secrets-edit` | Found 2026-08-20: three files under `~/.local/state/nvim/undodir/` holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664. Shredded. `make secrets-edit` now hardens `$EDITOR` before handing it plaintext, so it cannot recur. Never committed and never left the host, so those three communities were not rotated on that basis. | | Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. | | Pre-purge objects still served by GitHub after the history rewrite | The 2026-08-19 rewrite (`021d2b6`) removed both secrets above from every *reachable* commit, but GitHub still serves the orphaned objects by SHA. Verified 2026-08-26: `647d90a`, `21afcad`, `efb2632` and `ee3d443` all still resolve through the API, and the tree at `21afcad` still lists `certificates/Gandalf.Gondor.Lab/ca-key.pem` and `cert-key.pem`. Garbage collection requested from GitHub Support on 2026-08-26 — **pending**; this is the [purge runbook](docs/runbooks/purge-git-history.md)'s *Afterwards* step, and it is the last one outstanding. The repository has no forks and a network count of 0, so nothing else is perpetuating them. Both credentials were rotated *before* the rewrite, so this changes nothing about their status: the old keys and the old community remain superseded and must still be treated as public. Re-check with `gh api repos/Gerrrt/HomeLab/commits/647d90a --jq .sha` — a `404` means GitHub has collected them. | +| Prometheus, Loki and Alertmanager published on `0.0.0.0` with no authentication | Fixed 2026-08-30 — all three now bind to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md)). Firewall default-deny had been the only control, so anything that could route to `10.0.99.20` could read every metric and log line, inject metrics through Prometheus' remote-write receiver, delete log ranges through Loki's delete API, and silence any alert — the last of those without leaving a record anywhere the silence did not also suppress. Not reachable from an untrusted segment at any point: only Hicks (50) can reach Winterfell (99) at all. `.env.example` shipped the `0.0.0.0` default and `make up` regenerates `.env` from it, so that was the deployed value on every deploy. Grafana and the syslog receiver stay published; the first authenticates, and the second is a UDP receiver pfSense has to reach. | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is diff --git a/docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md b/docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md new file mode 100644 index 0000000..29bfb78 --- /dev/null +++ b/docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md @@ -0,0 +1,87 @@ +# ADR-0012: Bind the unauthenticated ports to loopback + +**Status:** Accepted · 2026-08 + +## Context + +The observability stack published four services to every VLAN the monitoring +host can reach, because a single `BIND_ADDR` in `.env.example` governed all of +them and shipped as `0.0.0.0`. Only one of the four authenticates. + +The other three do not, and each has a write surface: + +- **Prometheus** runs with `--web.enable-remote-write-receiver`, so anything + that can reach 9090 can inject metrics, and `--web.enable-lifecycle`, so it + can force a config reload. +- **Loki** runs `auth_enabled: false`, which is correct for a single-tenant + store but leaves the push and delete APIs open to whoever arrives. +- **Alertmanager** lets any caller create a silence. + +The last is the one that decided this. Silencing an alert is a quiet way to +switch off monitoring, and the record of the act lives in the system being +switched off. It is the only one of the three whose abuse is designed to leave +no trace. + +[ADR-0002](0002-vlan-segmentation-strategy.md)'s default-deny meant only Hicks +(50) could reach Winterfell (99) at all, so this was never exploitable from an +untrusted segment. But that made the firewall the entire control with nothing +behind it, and ADR-0002 itself already records the relevant weakness: *"A +compromised workstation reaches Winterfell."* One rule stood between an +ordinary desktop compromise and write access to the metric and log stores. + +Investigating turned up the fact that made this cheap to fix. **Nothing outside +this host consumes those three services.** Prometheus's scrape targets, +Grafana's datasources, Alertmanager's ruler URL and the local Alloy's push +endpoints all address services by compose name on the internal network, never by +a published host port. `morpheus` reaches Loki through Alloy's syslog receiver +rather than directly, and per +[#88](https://github.com/Gerrrt/HomeLab/issues/88) no other host runs an agent +yet. The wide bind was buying nothing that was actually being used. + +Two ports do have off-host clients that cannot be argued away: Grafana, opened +in a browser from Hicks, and Alloy's syslog receiver, which pfSense pushes to. + +## Decision + +Exposure is earned per service, not granted by a shared default. + +Prometheus, Alertmanager and Loki bind to `127.0.0.1`, hard-coded in +`compose.yaml` beside the Alloy debug UI that already did so. `BIND_ADDR` +survives, now governing only the two ports that earned it — Grafana, which +earns it by authenticating and being the one UI meant for a human, and the +syslog receiver, which earns it because pfSense cannot reach loopback. + +Loopback is hard-coded rather than made a second variable. `.env` is regenerated +from `.env.example` on every `make up`, so a variable would advertise a runtime +knob the deploy path does not actually provide; and widening one of these ports +hands a whole VLAN write access to the stores, which belongs in a reviewed diff +that trips the pull request template's *"No new port published to a VLAN that +could not already reach the service"* checkbox. + +No service flags change. `--web.enable-remote-write-receiver` is what #88 will +need, `--web.enable-lifecycle` is what `scripts/reload-config.sh` uses from +inside the container, and Loki's `auth_enabled: false` is correct for one +tenant. Behind a loopback bind none of the three is an exposure, and removing +them would cost capability to buy nothing. + +## Consequences + +- The firewall is no longer the only control. A compromised host on Hicks can + still reach Grafana, which asks it for a password. +- **Deploying an agent to another host (#88) now has a prerequisite.** The bind + must be widened in `compose.yaml` before Alloy on `Saruman` or `oracle` can + remote-write, and the firewall rule alone is no longer sufficient. + `runbooks/add-monitored-device.md` states both halves; without that, the + failure is an agent logging connection-refused against a host that answers + ping. +- Alertmanager's `--web.external-url` still points at `10.0.99.20:9093`, so the + `externalURL` link carried in webhook payloads now only opens on the + monitoring host. It is left pointing at the real address because that is + where the service is. Off-host, silences are in Grafana under Alerting, which + proxies Alertmanager over the compose network behind a login; on-host, + `docker exec alertmanager amtool` still works. +- The runbooks were already written from the monitoring host, so their + `curl localhost:...` verification steps are unaffected. +- This is a smaller decision than ADR-0002 and does not supersede it. It fills + in the layer that ADR-0002's consequences section admitted was missing, at a + scale where a bastion is still not worth building. diff --git a/docs/architecture.md b/docs/architecture.md index 0cf6f25..4394521 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -175,19 +175,33 @@ re-shard of everything. Reasoning in | Service | Port | Bound to | Notes | | --- | --- | --- | --- | -| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human | -| Prometheus | 9090 | `${BIND_ADDR}` | Also the remote-write receiver for agents | -| Loki | 3100 | `${BIND_ADDR}` | Push endpoint for agents | -| Alertmanager | 9093 | `${BIND_ADDR}` | | +| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only published service that authenticates | +| Prometheus | 9090 | `127.0.0.1` | Unauthenticated, and a remote-write receiver — reachable off-host it is a metric-injection endpoint | +| Loki | 3100 | `127.0.0.1` | Unauthenticated push and delete APIs (`auth_enabled: false`) | +| Alertmanager | 9093 | `127.0.0.1` | Unauthenticated — anyone who can reach it can silence an alert | | Alloy | 12345 | `127.0.0.1` | Debug UI, deliberately not exposed | | Alloy syslog | 1514/udp | `${BIND_ADDR}` | Network syslog receiver — pfSense pushes here | | snmp-exporter | 9116 | *compose network only* | Never published to a host interface | | blackbox-exporter | 9115 | *compose network only* | Never published — an open prober is an SSRF primitive | -`BIND_ADDR` defaults to `0.0.0.0` and is set in `.env`. Setting it to the host's -VLAN 99 address confines the whole stack to the management segment; the -published ports exist because agents on other hosts need to reach Prometheus and -Loki. +`BIND_ADDR` governs Grafana and the syslog receiver, and nothing else. It +defaults to `0.0.0.0` and is set in `stacks/observability/.env.example` — +`.env` is regenerated from that file on every `make up`, so the committed value +is the deployed one. Setting it to the host's VLAN 99 address confines those two +to the management segment. + +The other three bind to loopback deliberately +([ADR-0012](adr/0012-bind-the-unauthenticated-ports-to-loopback.md)). None of +them authenticates, and none has a consumer outside this host: Prometheus's +scrape targets, Grafana's datasources and the local Alloy all address services +by compose name on the internal network. Default-deny between VLANs is a real +control, but it was the only one, and a compromised workstation on Hicks sits +behind it. + +Widening one is a `compose.yaml` edit, not a setting. That is the point — an +agent on another host pushing to Prometheus or Loki +([#88](https://github.com/Gerrrt/HomeLab/issues/88)) grants a VLAN write access +to the stores, which is a change that should arrive as a reviewed diff. ## Reference diagrams diff --git a/docs/observability.md b/docs/observability.md index aedc223..45a0215 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -205,9 +205,10 @@ turned on itself. See [`runbooks/add-monitored-device.md`](runbooks/add-monitored-device.md). In short: -- **A Linux host:** run Alloy with `LOKI_URL` and - `PROMETHEUS_REMOTE_WRITE_URL` pointed at `10.0.99.20`. Nothing on the - monitoring host changes. +- **A Linux host:** widen the Prometheus and Loki binds in `compose.yaml` + first — they are `127.0.0.1` by default (#70) — then run Alloy with + `LOKI_URL` and `PROMETHEUS_REMOTE_WRITE_URL` pointed at `10.0.99.20`. That + bind change is the only thing on the monitoring host that has to move. - **An SNMP device:** append a target to `prometheus/targets/snmp.yaml` and a module plus auth to `snmp-exporter/generator.yaml`. file_sd picks the target up within five diff --git a/docs/runbooks/add-monitored-device.md b/docs/runbooks/add-monitored-device.md index 94aa6d1..26c8552 100644 --- a/docs/runbooks/add-monitored-device.md +++ b/docs/runbooks/add-monitored-device.md @@ -6,8 +6,23 @@ Two paths, depending on whether the device can run an agent. ## A Linux host -Nothing on the monitoring host changes. Alloy pushes; Prometheus does not need -to be told the host exists. +Alloy pushes, so Prometheus does not need to be told the host exists. But the +monitoring host is not untouched, and this is the step that is easy to miss. + +**First, publish the ports.** Prometheus and Loki bind to `127.0.0.1` +([ADR-0012](../adr/0012-bind-the-unauthenticated-ports-to-loopback.md)), so +until you change that an agent anywhere else gets connection-refused from a host +that answers ping — and Alloy retries a failed push with backoff, so the symptom +is silence rather than an error you will notice. Change the two `ports:` lines +in `stacks/observability/compose.yaml` from `127.0.0.1` to +`${BIND_ADDR:-0.0.0.0}`, update the ports table in +[`architecture.md`](../architecture.md) to match — CI checks that the two agree +— then `make up` and confirm with `ss -ltn`. + +Neither service authenticates, so this hands write access to the metric and log +stores to every host on whichever segment can now reach them. That is the trade +the loopback bind exists to make you state out loud; keep `BIND_ADDR` at the +management IP rather than `0.0.0.0` if VLAN 99 is all you need. On the new host. **These assume `sudo`** — if the account you have is in the `docker` group but has no sudo (which is the case for `atropos` on `oracle`), @@ -141,9 +156,9 @@ up{instance=~".*.*"} The host appears on the Host Overview and Logs dashboards automatically — both template their host variable from live label values. -**Firewall:** the new host must be able to reach `10.0.99.20` on 9090 and 3100. -If it is not on VLAN 99 or 50, that is a rule you have to add, and one worth -thinking about before you do. +**Firewall:** with the ports published, the new host must also be able to reach +`10.0.99.20` on 9090 and 3100. If it is not on VLAN 99 or 50, that is a rule you +have to add, and one worth thinking about before you do. --- diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 10f0aa5..69fb62c 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -54,6 +54,16 @@ curl -s localhost:9093/-/healthy # Alertmanager curl -sk https://localhost:3000/api/health # Grafana (-k: lab CA) ``` +The four above answer on `localhost` because that is where they are bound. +Confirm the binds themselves are what `docs/architecture.md` claims (#70) — +`ss` is the check, not the `curl`s, which pass either way: + +```bash +ss -ltn | grep -E ':(9090|3100|9093|12345)' # 127.0.0.1 on every line +ss -ltn | grep ':3000' # BIND_ADDR — 0.0.0.0 by default +ss -lun | grep ':1514' # BIND_ADDR +``` + Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index b4e1f4f..86b1001 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -344,6 +344,10 @@ devices. It also means published ports do not route, so reach the services with `docker exec` rather than from the host — and it is why `GF_INSTALL_PLUGINS` has to be emptied above. +`BIND_ADDR=127.0.0.1` in the export below is belt-and-braces on top of that, and +since #70 it covers only the rehearsal Grafana and syslog receiver — the other +three are pinned to loopback in `compose.yaml` and ignore it. + ```bash export COMPOSE_PROJECT_NAME=rehearse BIND_ADDR=127.0.0.1 \ PROMETHEUS_PORT=19090 ALERTMANAGER_PORT=19093 LOKI_PORT=13100 \ diff --git a/docs/security.md b/docs/security.md index 04b6d4f..4dce697 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,6 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs | +| Someone on a reachable VLAN reading, poisoning or silencing the monitoring stack | Prometheus, Loki and Alertmanager bind to `127.0.0.1`; only Grafana, which authenticates, is published | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | What it explicitly does **not** defend against: a determined attacker with @@ -67,6 +68,14 @@ one that admits the exception. Everything else — IoT, media, guest — gets internet and nothing more. +Segmentation is a control, not the only one. Until 2026-08 it was the only thing +standing between a workstation on Hicks and unauthenticated write access to the +metric and log stores — which is exactly the failure ADR-0002 predicted when it +recorded that "a compromised workstation reaches Winterfell". The stack's +unauthenticated ports now bind to loopback, so reaching VLAN 99 no longer buys +that; see +[ADR-0012](adr/0012-bind-the-unauthenticated-ports-to-loopback.md). + The IoT segment is the one that justifies the whole exercise. It holds cameras, a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya white-noise machine. Every one of those is a network-connected computer running @@ -221,6 +230,11 @@ complete state table and interface topology. They are credentials. its own unprivileged UID. - `snmp-exporter` is never published to a host interface — it is reachable only on the compose network. +- Prometheus, Alertmanager and Loki bind to `127.0.0.1` only. None of the three + authenticates, and each has a write surface — metric injection, silence + creation, log push and delete — so reaching VLAN 99 is no longer sufficient to + write to the stores. Nothing off the monitoring host consumed them; see + [ADR-0012](adr/0012-bind-the-unauthenticated-ports-to-loopback.md). - The Alloy debug UI binds to `127.0.0.1` only. - The Docker socket is mounted read-only into Alloy. - All images are pinned to explicit versions, so an upstream compromise cannot diff --git a/stacks/observability/.env.example b/stacks/observability/.env.example index 68085fe..34f0f2c 100644 --- a/stacks/observability/.env.example +++ b/stacks/observability/.env.example @@ -1,19 +1,31 @@ # Non-sensitive tunables for the observability stack. # -# Copy to .env and edit, or just rely on the defaults baked into compose.yaml. +# Edit this file, not .env — scripts/render-config.sh regenerates .env from it +# on every `make up`. Or rely on the defaults baked into compose.yaml. # Secrets do NOT belong here — see secrets/README.md. # # scripts/render-config.sh appends the decrypted secret values to .env at # deploy time, which is why .env itself is gitignored. -# Address the published ports bind to. 0.0.0.0 exposes the stack to every VLAN -# the host can reach; set this to the management IP to restrict it to VLAN 99. +# Address Grafana and the syslog receiver bind to. Those two are the only ports +# this governs (#70) — they are the only ones with an off-host client, a browser +# on Hicks and pfSense respectively. Prometheus, Alertmanager and Loki have no +# authentication and are pinned to 127.0.0.1 in compose.yaml; widening one is an +# edit there, deliberately, because it hands write access to a whole VLAN. +# +# 0.0.0.0 exposes these two to every VLAN the host can reach; set it to the +# management IP to restrict them to VLAN 99. Change it *here*: scripts/render- +# config.sh rewrites .env from this file on every `make up`, so an edit to .env +# lasts until the next deploy and no longer. BIND_ADDR=0.0.0.0 -# Used for Alertmanager's external URL in notification links. +# Used for Alertmanager's external URL in notification links. Since 9093 binds +# to loopback (#70) that link only opens on this host; off-host, silences are in +# Grafana under Alerting. MONITORING_HOST=10.0.99.20 -# Published ports. +# Host ports. PROMETHEUS, ALERTMANAGER, LOKI and ALLOY are published on +# 127.0.0.1; GRAFANA and SYSLOG on BIND_ADDR. See the note above. PROMETHEUS_PORT=9090 ALERTMANAGER_PORT=9093 LOKI_PORT=3100 diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 8891218..9f3ea54 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -8,18 +8,25 @@ make up # from the repository root | Service | Image | Port | Purpose | | --- | --- | --- | --- | -| `prometheus` | `prom/prometheus` | 9090 | Metrics store, remote-write receiver, rule evaluation | -| `alertmanager` | `prom/alertmanager` | 9093 | Alert routing, grouping, inhibition | -| `loki` | `grafana/loki` | 3100 | Log store | -| `grafana` | `grafana/grafana-oss` | 3000 | Dashboards | +| `prometheus` | `prom/prometheus` | 9090 (localhost) | Metrics store, remote-write receiver, rule evaluation | +| `alertmanager` | `prom/alertmanager` | 9093 (localhost) | Alert routing, grouping, inhibition | +| `loki` | `grafana/loki` | 3100 (localhost) | Log store | +| `grafana` | `grafana/grafana-oss` | 3000 | Dashboards — the only published UI, and the only one that authenticates | | `snmp-exporter` | `prom/snmp-exporter` | *internal* | SNMP polling proxy | | `alloy` | `grafana/alloy` | 12345 (localhost) | Metric and log collection | +"(localhost)" means bound to `127.0.0.1` on the monitoring host: reachable from +the host itself and over the compose network, and from no VLAN at all (#70). +Grafana and the Alloy syslog receiver (1514/udp, not in the table) are the two +that bind to `BIND_ADDR`. Reasoning in +[`docs/architecture.md`](../../docs/architecture.md#ports). + ## Layout ```text compose.yaml all seven services, one network, health-gated ordering .env.example non-sensitive tunables (ports, retention, bind address) + edit this, not .env — .env is regenerated on `make up` prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index bc52b71..59fb5d8 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -23,8 +23,10 @@ x-service-defaults: &service-defaults services: # --------------------------------------------------------------------------- - # Metrics store. Also acts as the remote-write sink for Alloy agents running - # on other hosts, hence --web.enable-remote-write-receiver. + # Metrics store, and the remote-write sink for Alloy — today only this host's + # own agent, reaching `prometheus` over the compose network, which is why 9090 + # is published to loopback only (#70). --web.enable-remote-write-receiver + # stays: the local agent uses it, and #88 will need it. # --------------------------------------------------------------------------- prometheus: <<: *service-defaults @@ -46,7 +48,13 @@ services: - ./prometheus/targets:/etc/prometheus/targets:ro - prometheus-data:/prometheus ports: - - "${BIND_ADDR:-0.0.0.0}:${PROMETHEUS_PORT:-9090}:9090" + # Loopback only (#70). Prometheus has no authentication, and with + # --web.enable-remote-write-receiver above, anything that can reach this + # port can inject metrics. Nothing off this host consumes it today: the + # scrape targets, Grafana's datasource and the local Alloy all address + # `prometheus` on the compose network. Deploying an agent elsewhere + # (#88) means widening this line deliberately, not editing .env. + - "127.0.0.1:${PROMETHEUS_PORT:-9090}:9090" healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"] interval: 30s @@ -70,6 +78,10 @@ services: command: - --config.file=/etc/alertmanager/alertmanager.yaml - --storage.path=/alertmanager + # Seeds the externalURL field in webhook payloads and nothing else — no + # receiver here renders it. Since #70 that link opens only on the + # monitoring host. Left pointing at the real address rather than + # 127.0.0.1, because that is still the host you would tunnel to. - --web.external-url=http://${MONITORING_HOST:-10.0.99.20}:${ALERTMANAGER_PORT:-9093} volumes: - ./alertmanager/alertmanager.yaml:/etc/alertmanager/alertmanager.yaml:ro @@ -79,7 +91,12 @@ services: - ./alertmanager/.rendered:/etc/alertmanager/secrets:ro - alertmanager-data:/alertmanager ports: - - "${BIND_ADDR:-0.0.0.0}:${ALERTMANAGER_PORT:-9093}:9093" + # Loopback only (#70). Anyone who can reach an unauthenticated + # Alertmanager can create a silence, which switches off an alert using + # the same system that would otherwise have recorded the act. Silences + # remain reachable off-host through Grafana, which proxies this service + # over the compose network behind a login. + - "127.0.0.1:${ALERTMANAGER_PORT:-9093}:9093" healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:9093/-/healthy"] interval: 30s @@ -105,7 +122,12 @@ services: - ./loki/rules:/etc/loki/rules/fake:ro - loki-data:/loki ports: - - "${BIND_ADDR:-0.0.0.0}:${LOKI_PORT:-3100}:3100" + # Loopback only (#70). auth_enabled is false, so the push and delete + # APIs are open to anything that can reach the port. The firewall logs + # arrive via Alloy's syslog receiver rather than here, and the local + # Alloy pushes to `loki` on the compose network — so, as with Prometheus, + # nothing off this host needs it until #88. + - "127.0.0.1:${LOKI_PORT:-3100}:3100" # No healthcheck on loki, deliberately. # @@ -265,6 +287,9 @@ services: group_add: - "${RENDER_GID:?run make render}" ports: + # One of only two ports that bind wide (#70), and the only one a human is + # meant to open. It earns that by authenticating: anonymous access and + # sign-up are both off, and the admin password comes from SOPS. - "${BIND_ADDR:-0.0.0.0}:${GRAFANA_PORT:-3000}:3000" depends_on: prometheus: @@ -335,6 +360,10 @@ services: # Unlike the debug UI above this must be reachable off-host, so it binds # to BIND_ADDR. 1514 rather than 514: the container does not run as root # and cannot take a privileged port. pfSense sends UDP by default. + # + # This and Grafana are now the only two ports BIND_ADDR governs (#70). + # Both have an off-host client that cannot be argued away — pfSense here, + # a browser on Hicks there. The rest bind to loopback. - "${BIND_ADDR:-0.0.0.0}:${SYSLOG_PORT:-1514}:1514/udp" depends_on: # service_started for loki, for the same reason as grafana above: the From b78d48171ac3a7ef15a544ee79e63085fd24d5ed Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 30 Aug 2026 23:41:45 +0000 Subject: [PATCH 2/2] =?UTF-8?q?fix(observability):=20narrow=20to=20Alertma?= =?UTF-8?q?nager=20=E2=80=94=20oracle=20is=20a=20live=20client=20(#70)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The premise of the previous commit was wrong. It claimed nothing off-host consumed Prometheus or Loki, on the strength of roadmap #88 saying only the monitoring host ran an agent. Checking the host rather than the document: `oracle` has been remote-writing to 10.0.99.20:9090 and pushing to :3100 for about eighteen hours — 378 distinct metric names, 42 log lines in ten minutes, and it is not a scrape target, so those published ports are its only path. Loopback-binding them would have cut both silently, since Alloy retries with backoff and buffers rather than failing loudly. So Prometheus and Loki go back to ${BIND_ADDR} and are recorded as an accepted residual instead: published, unauthenticated, with firewall default-deny the only control. Closing that needs authentication in front of the ingest ports and a credential on every agent, which is separate work. Alertmanager keeps the loopback bind, and it was always the sharpest of the three — a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port; Grafana proxies it over the compose network behind a login, so this costs no capability. Also drops the plan to narrow BIND_ADDR to the management IP. That rested on the host being a 2012 MacBook Pro and therefore having Wi-Fi; it has one interface, a USB ethernet adapter already on VLAN 99, so 0.0.0.0 and 10.0.99.20 are the same thing here. ADR-0012 is rewritten around the rule that actually survived — publish a port when something off-host uses it — and renamed accordingly. Roadmap #88 is corrected to reflect that oracle already has an agent. Co-Authored-By: Claude Opus 5 --- README.md | 5 +- SECURITY.md | 3 +- ...d-the-unauthenticated-ports-to-loopback.md | 87 ------------------ ...sh-only-ports-with-an-off-host-consumer.md | 90 +++++++++++++++++++ docs/architecture.md | 44 ++++----- docs/observability.md | 7 +- docs/roadmap.md | 7 +- docs/runbooks/add-monitored-device.md | 25 ++---- docs/runbooks/deploy-stack.md | 11 ++- docs/runbooks/restore-the-stack.md | 6 +- docs/security.md | 32 ++++--- stacks/observability/.env.example | 21 +++-- stacks/observability/README.md | 13 +-- stacks/observability/compose.yaml | 50 +++++------ 14 files changed, 194 insertions(+), 207 deletions(-) delete mode 100644 docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md create mode 100644 docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md diff --git a/README.md b/README.md index 7532774..de6a425 100644 --- a/README.md +++ b/README.md @@ -195,9 +195,8 @@ Prometheus verifies it with the CA, so `make up` renders nothing until they exist. Details in [`docs/runbooks/generate-certificates.md`](docs/runbooks/generate-certificates.md). -Grafana on `:3000` over https, from any workstation that can route to the host. -Prometheus, Loki and Alertmanager bind to `127.0.0.1` and are reached from the -host itself or through Grafana +Grafana on `:3000` over https, Prometheus on `:9090`. Alertmanager binds to +`127.0.0.1` and is reached through Grafana ([#70](https://github.com/Gerrrt/HomeLab/issues/70)). Grafana's certificate is signed by the lab's own CA, so a browser warns and `curl` needs `-k` until you trust `certificates/ca.pem` — step 4 of that runbook. Full procedure, diff --git a/SECURITY.md b/SECURITY.md index 11c44fa..94def5b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -44,7 +44,8 @@ is a very different thing from an overlooked one. Full detail in | Decrypted secrets in editor undo files, written by `make secrets-edit` | Found 2026-08-20: three files under `~/.local/state/nvim/undodir/` holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664. Shredded. `make secrets-edit` now hardens `$EDITOR` before handing it plaintext, so it cannot recur. Never committed and never left the host, so those three communities were not rotated on that basis. | | Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. | | Pre-purge objects still served by GitHub after the history rewrite | The 2026-08-19 rewrite (`021d2b6`) removed both secrets above from every *reachable* commit, but GitHub still serves the orphaned objects by SHA. Verified 2026-08-26: `647d90a`, `21afcad`, `efb2632` and `ee3d443` all still resolve through the API, and the tree at `21afcad` still lists `certificates/Gandalf.Gondor.Lab/ca-key.pem` and `cert-key.pem`. Garbage collection requested from GitHub Support on 2026-08-26 — **pending**; this is the [purge runbook](docs/runbooks/purge-git-history.md)'s *Afterwards* step, and it is the last one outstanding. The repository has no forks and a network count of 0, so nothing else is perpetuating them. Both credentials were rotated *before* the rewrite, so this changes nothing about their status: the old keys and the old community remain superseded and must still be treated as public. Re-check with `gh api repos/Gerrrt/HomeLab/commits/647d90a --jq .sha` — a `404` means GitHub has collected them. | -| Prometheus, Loki and Alertmanager published on `0.0.0.0` with no authentication | Fixed 2026-08-30 — all three now bind to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md)). Firewall default-deny had been the only control, so anything that could route to `10.0.99.20` could read every metric and log line, inject metrics through Prometheus' remote-write receiver, delete log ranges through Loki's delete API, and silence any alert — the last of those without leaving a record anywhere the silence did not also suppress. Not reachable from an untrusted segment at any point: only Hicks (50) can reach Winterfell (99) at all. `.env.example` shipped the `0.0.0.0` default and `make up` regenerates `.env` from it, so that was the deployed value on every deploy. Grafana and the syslog receiver stay published; the first authenticates, and the second is a UDP receiver pfSense has to reach. | +| Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. | +| Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control: only Hicks (50) and Winterfell (99) can reach the host at all, and never an untrusted segment. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is diff --git a/docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md b/docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md deleted file mode 100644 index 29bfb78..0000000 --- a/docs/adr/0012-bind-the-unauthenticated-ports-to-loopback.md +++ /dev/null @@ -1,87 +0,0 @@ -# ADR-0012: Bind the unauthenticated ports to loopback - -**Status:** Accepted · 2026-08 - -## Context - -The observability stack published four services to every VLAN the monitoring -host can reach, because a single `BIND_ADDR` in `.env.example` governed all of -them and shipped as `0.0.0.0`. Only one of the four authenticates. - -The other three do not, and each has a write surface: - -- **Prometheus** runs with `--web.enable-remote-write-receiver`, so anything - that can reach 9090 can inject metrics, and `--web.enable-lifecycle`, so it - can force a config reload. -- **Loki** runs `auth_enabled: false`, which is correct for a single-tenant - store but leaves the push and delete APIs open to whoever arrives. -- **Alertmanager** lets any caller create a silence. - -The last is the one that decided this. Silencing an alert is a quiet way to -switch off monitoring, and the record of the act lives in the system being -switched off. It is the only one of the three whose abuse is designed to leave -no trace. - -[ADR-0002](0002-vlan-segmentation-strategy.md)'s default-deny meant only Hicks -(50) could reach Winterfell (99) at all, so this was never exploitable from an -untrusted segment. But that made the firewall the entire control with nothing -behind it, and ADR-0002 itself already records the relevant weakness: *"A -compromised workstation reaches Winterfell."* One rule stood between an -ordinary desktop compromise and write access to the metric and log stores. - -Investigating turned up the fact that made this cheap to fix. **Nothing outside -this host consumes those three services.** Prometheus's scrape targets, -Grafana's datasources, Alertmanager's ruler URL and the local Alloy's push -endpoints all address services by compose name on the internal network, never by -a published host port. `morpheus` reaches Loki through Alloy's syslog receiver -rather than directly, and per -[#88](https://github.com/Gerrrt/HomeLab/issues/88) no other host runs an agent -yet. The wide bind was buying nothing that was actually being used. - -Two ports do have off-host clients that cannot be argued away: Grafana, opened -in a browser from Hicks, and Alloy's syslog receiver, which pfSense pushes to. - -## Decision - -Exposure is earned per service, not granted by a shared default. - -Prometheus, Alertmanager and Loki bind to `127.0.0.1`, hard-coded in -`compose.yaml` beside the Alloy debug UI that already did so. `BIND_ADDR` -survives, now governing only the two ports that earned it — Grafana, which -earns it by authenticating and being the one UI meant for a human, and the -syslog receiver, which earns it because pfSense cannot reach loopback. - -Loopback is hard-coded rather than made a second variable. `.env` is regenerated -from `.env.example` on every `make up`, so a variable would advertise a runtime -knob the deploy path does not actually provide; and widening one of these ports -hands a whole VLAN write access to the stores, which belongs in a reviewed diff -that trips the pull request template's *"No new port published to a VLAN that -could not already reach the service"* checkbox. - -No service flags change. `--web.enable-remote-write-receiver` is what #88 will -need, `--web.enable-lifecycle` is what `scripts/reload-config.sh` uses from -inside the container, and Loki's `auth_enabled: false` is correct for one -tenant. Behind a loopback bind none of the three is an exposure, and removing -them would cost capability to buy nothing. - -## Consequences - -- The firewall is no longer the only control. A compromised host on Hicks can - still reach Grafana, which asks it for a password. -- **Deploying an agent to another host (#88) now has a prerequisite.** The bind - must be widened in `compose.yaml` before Alloy on `Saruman` or `oracle` can - remote-write, and the firewall rule alone is no longer sufficient. - `runbooks/add-monitored-device.md` states both halves; without that, the - failure is an agent logging connection-refused against a host that answers - ping. -- Alertmanager's `--web.external-url` still points at `10.0.99.20:9093`, so the - `externalURL` link carried in webhook payloads now only opens on the - monitoring host. It is left pointing at the real address because that is - where the service is. Off-host, silences are in Grafana under Alerting, which - proxies Alertmanager over the compose network behind a login; on-host, - `docker exec alertmanager amtool` still works. -- The runbooks were already written from the monitoring host, so their - `curl localhost:...` verification steps are unaffected. -- This is a smaller decision than ADR-0002 and does not supersede it. It fills - in the layer that ADR-0002's consequences section admitted was missing, at a - scale where a bastion is still not worth building. diff --git a/docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md b/docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md new file mode 100644 index 0000000..3e8b428 --- /dev/null +++ b/docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md @@ -0,0 +1,90 @@ +# ADR-0012: Publish only the ports something off-host uses + +**Status:** Accepted · 2026-08 + +## Context + +The observability stack published five ports on `${BIND_ADDR:-0.0.0.0}` because +one variable governed all of them. Only Grafana authenticates. The others each +carry a write surface: Prometheus runs with `--web.enable-remote-write-receiver` +and `--web.enable-lifecycle`, Loki runs `auth_enabled: false` with a delete API, +and Alertmanager lets any caller create a silence. + +The last of those is the one that prompted this. Silencing an alert switches off +monitoring, and the record of the act lives in the system being switched off. It +is the only one of the three whose abuse is designed to leave no trace. + +[ADR-0002](0002-vlan-segmentation-strategy.md)'s default-deny means only Hicks +(50) can reach Winterfell (99), so none of this was ever exposed to an untrusted +segment. But it made one firewall rule the entire control, and ADR-0002 already +records the gap that leaves: *"A compromised workstation reaches Winterfell."* + +The question was whether the wide default was earning anything. Checking rather +than assuming turned out to matter, and in both directions: + +- `docs/roadmap.md` said only the monitoring host ran an agent. It was a day out + of date. `oracle` had been remote-writing to `10.0.99.20:9090` and pushing to + `:3100` for about eighteen hours — 378 distinct metric names, and it is not a + scrape target, so those published ports are its only path. Prometheus and Loki + are load-bearing. +- Alertmanager had no off-host client at all, and still does not. Grafana + proxies it over the compose network through an authenticated datasource, so + every operator path to a silence already avoided port 9093. + +A second assumption did not survive either: the monitoring host is a 2012 +MacBook Pro, so narrowing `BIND_ADDR` from `0.0.0.0` to the management IP looked +like cheap defence in depth. It is not — the machine has one interface, a USB +ethernet adapter already on VLAN 99. `0.0.0.0` and `10.0.99.20` are the same +thing here, and pinning the literal address would only add a way for the stack +to fail if it ever moved. + +## Decision + +A port is published to a host interface when something off the host uses it, and +not otherwise. The test is a named client, not a plausible future one. + +- **Grafana (3000)** — browsers on Hicks. Published, and the only published + service that authenticates. +- **Prometheus (9090)** and **Loki (3100)** — `oracle`'s Alloy agent. + Published. +- **Alloy syslog (1514/udp)** — `morpheus`. Published; pfSense cannot reach + loopback. +- **Alertmanager (9093)** — nothing. Bound to `127.0.0.1`, hard-coded beside the + Alloy debug UI (12345) that was already there. + +Loopback is hard-coded rather than made a variable. `.env` is regenerated from +`.env.example` on every `make up`, so a variable would advertise a runtime knob +the deploy path does not provide; and publishing one of these hands a VLAN write +access to a store, which should arrive as a reviewed diff against the pull +request template's *"No new port published to a VLAN that could not already +reach the service"* checkbox. + +No service flags change. `--web.enable-remote-write-receiver` is what `oracle` +uses, `--web.enable-lifecycle` is what `scripts/reload-config.sh` uses from +inside the container, and Loki's `auth_enabled: false` is correct for one +tenant. The flags were never the problem; reachability was. + +## Consequences + +- **The silence surface is closed.** The one write path whose abuse leaves no + trace is no longer reachable from any VLAN. +- **Prometheus and Loki remain an accepted residual, not a fixed one.** They are + published, unauthenticated, and writable by anything that can route to + `10.0.99.20` — which today means Hicks and Winterfell. The firewall is still + the only control there. Recorded in [`SECURITY.md`](../../SECURITY.md) rather + than quietly carried; closing it needs authentication in front of the ingest + ports, which is a separate piece of work and a new secret to rotate. +- Alertmanager's `--web.external-url` still points at `10.0.99.20:9093`, so the + `externalURL` field in webhook payloads now only opens on the monitoring host + or through an SSH tunnel. No receiver renders it, so no notification text + changes. Off-host, silences are in Grafana under Alerting; on-host, + `docker exec alertmanager amtool` still works. +- `127.0.0.1` is a host boundary, not a user one. Any local account on the + monitoring host, and any container on the host network, still reaches + Alertmanager unauthenticated. +- **The rule generalises to the next agent.** Deploying Alloy to `Saruman` + ([#88](https://github.com/Gerrrt/HomeLab/issues/88)) needs no bind change, + because `oracle` already made Prometheus and Loki published services. Adding a + service with no off-host client should default to loopback. +- Supersedes nothing. It fills in a layer ADR-0002's consequences section + admitted was missing, at a scale where a bastion is still not worth building. diff --git a/docs/architecture.md b/docs/architecture.md index 4394521..1e2e537 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -176,32 +176,34 @@ re-shard of everything. Reasoning in | Service | Port | Bound to | Notes | | --- | --- | --- | --- | | Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only published service that authenticates | -| Prometheus | 9090 | `127.0.0.1` | Unauthenticated, and a remote-write receiver — reachable off-host it is a metric-injection endpoint | -| Loki | 3100 | `127.0.0.1` | Unauthenticated push and delete APIs (`auth_enabled: false`) | -| Alertmanager | 9093 | `127.0.0.1` | Unauthenticated — anyone who can reach it can silence an alert | +| Prometheus | 9090 | `${BIND_ADDR}` | Remote-write receiver — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | +| Loki | 3100 | `${BIND_ADDR}` | Push endpoint — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | +| Alertmanager | 9093 | `127.0.0.1` | Nothing off-host uses it; silences are reached through Grafana | | Alloy | 12345 | `127.0.0.1` | Debug UI, deliberately not exposed | | Alloy syslog | 1514/udp | `${BIND_ADDR}` | Network syslog receiver — pfSense pushes here | | snmp-exporter | 9116 | *compose network only* | Never published to a host interface | | blackbox-exporter | 9115 | *compose network only* | Never published — an open prober is an SSRF primitive | -`BIND_ADDR` governs Grafana and the syslog receiver, and nothing else. It -defaults to `0.0.0.0` and is set in `stacks/observability/.env.example` — -`.env` is regenerated from that file on every `make up`, so the committed value -is the deployed one. Setting it to the host's VLAN 99 address confines those two -to the management segment. - -The other three bind to loopback deliberately -([ADR-0012](adr/0012-bind-the-unauthenticated-ports-to-loopback.md)). None of -them authenticates, and none has a consumer outside this host: Prometheus's -scrape targets, Grafana's datasources and the local Alloy all address services -by compose name on the internal network. Default-deny between VLANs is a real -control, but it was the only one, and a compromised workstation on Hicks sits -behind it. - -Widening one is a `compose.yaml` edit, not a setting. That is the point — an -agent on another host pushing to Prometheus or Loki -([#88](https://github.com/Gerrrt/HomeLab/issues/88)) grants a VLAN write access -to the stores, which is a change that should arrive as a reviewed diff. +A port is published only when something off this host uses it +([ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md)). Grafana +is opened in a browser from Hicks, the syslog receiver takes pushes from +`morpheus`, and Prometheus and Loki take metrics and logs from `oracle`'s Alloy +agent. Alertmanager has no such client, so it binds to `127.0.0.1`; silences are +reached through Grafana, which proxies it over the compose network behind a +login. + +`BIND_ADDR` governs the four that are published. It defaults to `0.0.0.0` and is +set in `stacks/observability/.env.example` — `.env` is regenerated from that +file on every `make up`, so the committed value is the deployed one. Setting it +to the host's VLAN 99 address would confine them to the management segment, +which today changes nothing: the host has one interface and it is already on +VLAN 99. + +Prometheus and Loki are published and unauthenticated, which is a real residual +rather than a solved problem — anything that can reach them can read every +metric and log line, inject metrics, and delete log ranges. Default-deny between +VLANs is the only control on that, and it is recorded as such in +[`SECURITY.md`](../SECURITY.md). ## Reference diagrams diff --git a/docs/observability.md b/docs/observability.md index 45a0215..aedc223 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -205,10 +205,9 @@ turned on itself. See [`runbooks/add-monitored-device.md`](runbooks/add-monitored-device.md). In short: -- **A Linux host:** widen the Prometheus and Loki binds in `compose.yaml` - first — they are `127.0.0.1` by default (#70) — then run Alloy with - `LOKI_URL` and `PROMETHEUS_REMOTE_WRITE_URL` pointed at `10.0.99.20`. That - bind change is the only thing on the monitoring host that has to move. +- **A Linux host:** run Alloy with `LOKI_URL` and + `PROMETHEUS_REMOTE_WRITE_URL` pointed at `10.0.99.20`. Nothing on the + monitoring host changes. - **An SNMP device:** append a target to `prometheus/targets/snmp.yaml` and a module plus auth to `snmp-exporter/generator.yaml`. file_sd picks the target up within five diff --git a/docs/roadmap.md b/docs/roadmap.md index e42ce36..d31c960 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -32,9 +32,10 @@ issues intact. Nothing was summarised away. seconds at gigabit. Now actionable — the switch has been polling since the faults in [#22](https://github.com/Gerrrt/HomeLab/issues/22) cleared. The walk has a trap in it; the issue carries the detail. -- **[#88](https://github.com/Gerrrt/HomeLab/issues/88) Deploy Alloy to `Saruman` - and `oracle`.** Only the monitoring host runs an agent; `morpheus` reaches Loki - by network syslog and has no host metrics. +- **[#88](https://github.com/Gerrrt/HomeLab/issues/88) Deploy Alloy to + `Saruman`.** `oracle` has had an agent since 2026-08-30 and is remote-writing + host metrics and pushing logs; `Saruman` is the one left. `morpheus` reaches + Loki by network syslog and has no host metrics. - **[#89](https://github.com/Gerrrt/HomeLab/issues/89) Extend Suricata to Degens (VLAN 10).** One interface at a time, once Skids has been quiet and understood for a few days. → [runbook](runbooks/enable-suricata.md) diff --git a/docs/runbooks/add-monitored-device.md b/docs/runbooks/add-monitored-device.md index 26c8552..94aa6d1 100644 --- a/docs/runbooks/add-monitored-device.md +++ b/docs/runbooks/add-monitored-device.md @@ -6,23 +6,8 @@ Two paths, depending on whether the device can run an agent. ## A Linux host -Alloy pushes, so Prometheus does not need to be told the host exists. But the -monitoring host is not untouched, and this is the step that is easy to miss. - -**First, publish the ports.** Prometheus and Loki bind to `127.0.0.1` -([ADR-0012](../adr/0012-bind-the-unauthenticated-ports-to-loopback.md)), so -until you change that an agent anywhere else gets connection-refused from a host -that answers ping — and Alloy retries a failed push with backoff, so the symptom -is silence rather than an error you will notice. Change the two `ports:` lines -in `stacks/observability/compose.yaml` from `127.0.0.1` to -`${BIND_ADDR:-0.0.0.0}`, update the ports table in -[`architecture.md`](../architecture.md) to match — CI checks that the two agree -— then `make up` and confirm with `ss -ltn`. - -Neither service authenticates, so this hands write access to the metric and log -stores to every host on whichever segment can now reach them. That is the trade -the loopback bind exists to make you state out loud; keep `BIND_ADDR` at the -management IP rather than `0.0.0.0` if VLAN 99 is all you need. +Nothing on the monitoring host changes. Alloy pushes; Prometheus does not need +to be told the host exists. On the new host. **These assume `sudo`** — if the account you have is in the `docker` group but has no sudo (which is the case for `atropos` on `oracle`), @@ -156,9 +141,9 @@ up{instance=~".*.*"} The host appears on the Host Overview and Logs dashboards automatically — both template their host variable from live label values. -**Firewall:** with the ports published, the new host must also be able to reach -`10.0.99.20` on 9090 and 3100. If it is not on VLAN 99 or 50, that is a rule you -have to add, and one worth thinking about before you do. +**Firewall:** the new host must be able to reach `10.0.99.20` on 9090 and 3100. +If it is not on VLAN 99 or 50, that is a rule you have to add, and one worth +thinking about before you do. --- diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 69fb62c..bba9779 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -54,14 +54,13 @@ curl -s localhost:9093/-/healthy # Alertmanager curl -sk https://localhost:3000/api/health # Grafana (-k: lab CA) ``` -The four above answer on `localhost` because that is where they are bound. -Confirm the binds themselves are what `docs/architecture.md` claims (#70) — -`ss` is the check, not the `curl`s, which pass either way: +The `curl`s pass wherever the services are bound, so confirm the binds +themselves are what `docs/architecture.md` claims (#70): ```bash -ss -ltn | grep -E ':(9090|3100|9093|12345)' # 127.0.0.1 on every line -ss -ltn | grep ':3000' # BIND_ADDR — 0.0.0.0 by default -ss -lun | grep ':1514' # BIND_ADDR +ss -ltn | grep -E ':(9093|12345)' # 127.0.0.1 — Alertmanager and Alloy +ss -ltn | grep -E ':(9090|3100|3000)' # BIND_ADDR — 0.0.0.0 by default +ss -lun | grep ':1514' # BIND_ADDR ``` Then in the UI: diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index 86b1001..eae28f5 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -344,9 +344,9 @@ devices. It also means published ports do not route, so reach the services with `docker exec` rather than from the host — and it is why `GF_INSTALL_PLUGINS` has to be emptied above. -`BIND_ADDR=127.0.0.1` in the export below is belt-and-braces on top of that, and -since #70 it covers only the rehearsal Grafana and syslog receiver — the other -three are pinned to loopback in `compose.yaml` and ignore it. +`BIND_ADDR=127.0.0.1` in the export below is belt-and-braces on top of that. It +covers the rehearsal Prometheus, Loki, Grafana and syslog receiver; Alertmanager +is pinned to loopback in `compose.yaml` and ignores it (#70). ```bash export COMPOSE_PROJECT_NAME=rehearse BIND_ADDR=127.0.0.1 \ diff --git a/docs/security.md b/docs/security.md index 4dce697..2b52868 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs | -| Someone on a reachable VLAN reading, poisoning or silencing the monitoring stack | Prometheus, Loki and Alertmanager bind to `127.0.0.1`; only Grafana, which authenticates, is published | +| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | What it explicitly does **not** defend against: a determined attacker with @@ -68,13 +68,18 @@ one that admits the exception. Everything else — IoT, media, guest — gets internet and nothing more. -Segmentation is a control, not the only one. Until 2026-08 it was the only thing -standing between a workstation on Hicks and unauthenticated write access to the -metric and log stores — which is exactly the failure ADR-0002 predicted when it -recorded that "a compromised workstation reaches Winterfell". The stack's -unauthenticated ports now bind to loopback, so reaching VLAN 99 no longer buys -that; see -[ADR-0012](adr/0012-bind-the-unauthenticated-ports-to-loopback.md). +Segmentation is doing more work here than it should have to. A workstation on +Hicks that can reach `10.0.99.20` can write to the metric and log stores without +a credential, because Prometheus and Loki publish unauthenticated ingest ports +for `oracle`'s agent to use — which is exactly the failure ADR-0002 predicted +when it recorded that "a compromised workstation reaches Winterfell". That is an +accepted residual, recorded in [`SECURITY.md`](../SECURITY.md), not a solved +problem. + +What has been taken off the firewall's shoulders is Alertmanager. It had no +off-host client, so it now binds to `127.0.0.1` and reaching VLAN 99 no longer +lets anyone silence an alert; see +[ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). The IoT segment is the one that justifies the whole exercise. It holds cameras, a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya @@ -230,11 +235,12 @@ complete state table and interface topology. They are credentials. its own unprivileged UID. - `snmp-exporter` is never published to a host interface — it is reachable only on the compose network. -- Prometheus, Alertmanager and Loki bind to `127.0.0.1` only. None of the three - authenticates, and each has a write surface — metric injection, silence - creation, log push and delete — so reaching VLAN 99 is no longer sufficient to - write to the stores. Nothing off the monitoring host consumed them; see - [ADR-0012](adr/0012-bind-the-unauthenticated-ports-to-loopback.md). +- Alertmanager binds to `127.0.0.1` only. It is unauthenticated, and a silence + is how monitoring gets switched off — quietly, since the record lives in the + system being switched off. Nothing off-host used the port; silences are + reached through Grafana. Prometheus and Loki are *not* in this list: they stay + published for `oracle`'s agent and remain an accepted residual. See + [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). - The Alloy debug UI binds to `127.0.0.1` only. - The Docker socket is mounted read-only into Alloy. - All images are pinned to explicit versions, so an upstream compromise cannot diff --git a/stacks/observability/.env.example b/stacks/observability/.env.example index 34f0f2c..a1ee1e5 100644 --- a/stacks/observability/.env.example +++ b/stacks/observability/.env.example @@ -7,16 +7,15 @@ # scripts/render-config.sh appends the decrypted secret values to .env at # deploy time, which is why .env itself is gitignored. -# Address Grafana and the syslog receiver bind to. Those two are the only ports -# this governs (#70) — they are the only ones with an off-host client, a browser -# on Hicks and pfSense respectively. Prometheus, Alertmanager and Loki have no -# authentication and are pinned to 127.0.0.1 in compose.yaml; widening one is an -# edit there, deliberately, because it hands write access to a whole VLAN. +# Address the published ports bind to. 0.0.0.0 exposes them to every VLAN the +# host can reach; set this to the management IP to restrict them to VLAN 99. # -# 0.0.0.0 exposes these two to every VLAN the host can reach; set it to the -# management IP to restrict them to VLAN 99. Change it *here*: scripts/render- -# config.sh rewrites .env from this file on every `make up`, so an edit to .env -# lasts until the next deploy and no longer. +# Change it *here*, not in .env: scripts/render-config.sh rewrites .env from +# this file on every `make up`, so an edit to .env lasts until the next deploy +# and no longer. The committed value is the deployed value. +# +# Alertmanager is not governed by this — it is pinned to 127.0.0.1 in +# compose.yaml, being the one service with no off-host client (#70). BIND_ADDR=0.0.0.0 # Used for Alertmanager's external URL in notification links. Since 9093 binds @@ -24,8 +23,8 @@ BIND_ADDR=0.0.0.0 # Grafana under Alerting. MONITORING_HOST=10.0.99.20 -# Host ports. PROMETHEUS, ALERTMANAGER, LOKI and ALLOY are published on -# 127.0.0.1; GRAFANA and SYSLOG on BIND_ADDR. See the note above. +# Host ports. ALERTMANAGER and ALLOY are published on 127.0.0.1; the rest on +# BIND_ADDR. See the note above. PROMETHEUS_PORT=9090 ALERTMANAGER_PORT=9093 LOKI_PORT=3100 diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 9f3ea54..05e401d 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -8,17 +8,18 @@ make up # from the repository root | Service | Image | Port | Purpose | | --- | --- | --- | --- | -| `prometheus` | `prom/prometheus` | 9090 (localhost) | Metrics store, remote-write receiver, rule evaluation | +| `prometheus` | `prom/prometheus` | 9090 | Metrics store, remote-write receiver, rule evaluation | | `alertmanager` | `prom/alertmanager` | 9093 (localhost) | Alert routing, grouping, inhibition | -| `loki` | `grafana/loki` | 3100 (localhost) | Log store | +| `loki` | `grafana/loki` | 3100 | Log store | | `grafana` | `grafana/grafana-oss` | 3000 | Dashboards — the only published UI, and the only one that authenticates | | `snmp-exporter` | `prom/snmp-exporter` | *internal* | SNMP polling proxy | | `alloy` | `grafana/alloy` | 12345 (localhost) | Metric and log collection | -"(localhost)" means bound to `127.0.0.1` on the monitoring host: reachable from -the host itself and over the compose network, and from no VLAN at all (#70). -Grafana and the Alloy syslog receiver (1514/udp, not in the table) are the two -that bind to `BIND_ADDR`. Reasoning in +"(localhost)" means bound to `127.0.0.1`: reachable from the monitoring host +itself and over the compose network, and from no VLAN at all. A port is +published only when something off-host uses it, and nothing off-host uses +Alertmanager (#70). The rest, plus the Alloy syslog receiver on 1514/udp, bind +to `BIND_ADDR`. Reasoning in [`docs/architecture.md`](../../docs/architecture.md#ports). ## Layout diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 59fb5d8..125217e 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -23,10 +23,8 @@ x-service-defaults: &service-defaults services: # --------------------------------------------------------------------------- - # Metrics store, and the remote-write sink for Alloy — today only this host's - # own agent, reaching `prometheus` over the compose network, which is why 9090 - # is published to loopback only (#70). --web.enable-remote-write-receiver - # stays: the local agent uses it, and #88 will need it. + # Metrics store. Also acts as the remote-write sink for Alloy agents running + # on other hosts, hence --web.enable-remote-write-receiver. # --------------------------------------------------------------------------- prometheus: <<: *service-defaults @@ -48,13 +46,12 @@ services: - ./prometheus/targets:/etc/prometheus/targets:ro - prometheus-data:/prometheus ports: - # Loopback only (#70). Prometheus has no authentication, and with - # --web.enable-remote-write-receiver above, anything that can reach this - # port can inject metrics. Nothing off this host consumes it today: the - # scrape targets, Grafana's datasource and the local Alloy all address - # `prometheus` on the compose network. Deploying an agent elsewhere - # (#88) means widening this line deliberately, not editing .env. - - "127.0.0.1:${PROMETHEUS_PORT:-9090}:9090" + # Published, and unauthenticated with it: anything that can reach this + # port can read every metric and inject more through the remote-write + # receiver. It stays published because `oracle` remote-writes here — that + # is a real client, not a hypothetical one (#70). Recorded as an accepted + # residual in SECURITY.md rather than left implicit. + - "${BIND_ADDR:-0.0.0.0}:${PROMETHEUS_PORT:-9090}:9090" healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"] interval: 30s @@ -91,11 +88,12 @@ services: - ./alertmanager/.rendered:/etc/alertmanager/secrets:ro - alertmanager-data:/alertmanager ports: - # Loopback only (#70). Anyone who can reach an unauthenticated - # Alertmanager can create a silence, which switches off an alert using - # the same system that would otherwise have recorded the act. Silences - # remain reachable off-host through Grafana, which proxies this service - # over the compose network behind a login. + # Loopback only, and the only one of the four that can be (#70). Anyone + # who can reach an unauthenticated Alertmanager can create a silence, + # which switches off an alert using the same system that would otherwise + # have recorded the act. Unlike Prometheus and Loki above, nothing + # off-host talks to it: silences are reached through Grafana, which + # proxies this service over the compose network behind a login. - "127.0.0.1:${ALERTMANAGER_PORT:-9093}:9093" healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:9093/-/healthy"] @@ -122,12 +120,10 @@ services: - ./loki/rules:/etc/loki/rules/fake:ro - loki-data:/loki ports: - # Loopback only (#70). auth_enabled is false, so the push and delete - # APIs are open to anything that can reach the port. The firewall logs - # arrive via Alloy's syslog receiver rather than here, and the local - # Alloy pushes to `loki` on the compose network — so, as with Prometheus, - # nothing off this host needs it until #88. - - "127.0.0.1:${LOKI_PORT:-3100}:3100" + # Published for the same reason as Prometheus above, and with the same + # residual: auth_enabled is false, so the push and delete APIs are open to + # whatever can reach the port. `oracle`'s Alloy pushes here (#70). + - "${BIND_ADDR:-0.0.0.0}:${LOKI_PORT:-3100}:3100" # No healthcheck on loki, deliberately. # @@ -287,9 +283,9 @@ services: group_add: - "${RENDER_GID:?run make render}" ports: - # One of only two ports that bind wide (#70), and the only one a human is - # meant to open. It earns that by authenticating: anonymous access and - # sign-up are both off, and the admin password comes from SOPS. + # The only published port a human is meant to open, and the only one that + # authenticates: anonymous access and sign-up are both off, and the admin + # password comes from SOPS. - "${BIND_ADDR:-0.0.0.0}:${GRAFANA_PORT:-3000}:3000" depends_on: prometheus: @@ -360,10 +356,6 @@ services: # Unlike the debug UI above this must be reachable off-host, so it binds # to BIND_ADDR. 1514 rather than 514: the container does not run as root # and cannot take a privileged port. pfSense sends UDP by default. - # - # This and Grafana are now the only two ports BIND_ADDR governs (#70). - # Both have an off-host client that cannot be argued away — pfSense here, - # a browser on Hicks there. The rest bind to loopback. - "${BIND_ADDR:-0.0.0.0}:${SYSLOG_PORT:-1514}:1514/udp" depends_on: # service_started for loki, for the same reason as grafana above: the