diff --git a/README.md b/README.md index b276415..de6a425 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ rack; a dashed border means egress only. Full topology and data flow in ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 11 architecture decision records +│ ├── adr/ # 12 architecture decision records │ └── runbooks/ # deploy, add device, rotate creds, certs, key backup, │ # purge, restore the firewall, restore the stack, │ # ship firewall logs, verify the alert path, @@ -195,7 +195,9 @@ Prometheus verifies it with the CA, so `make up` renders nothing until they exist. Details in [`docs/runbooks/generate-certificates.md`](docs/runbooks/generate-certificates.md). -Grafana on `:3000` over https, Prometheus on `:9090`. Grafana's certificate is +Grafana on `:3000` over https, Prometheus on `:9090`. Alertmanager binds to +`127.0.0.1` and is reached through Grafana +([#70](https://github.com/Gerrrt/HomeLab/issues/70)). Grafana's certificate is signed by the lab's own CA, so a browser warns and `curl` needs `-k` until you trust `certificates/ca.pem` — step 4 of that runbook. Full procedure, verification steps and troubleshooting in diff --git a/SECURITY.md b/SECURITY.md index 8ec89b5..94def5b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -44,6 +44,8 @@ is a very different thing from an overlooked one. Full detail in | Decrypted secrets in editor undo files, written by `make secrets-edit` | Found 2026-08-20: three files under `~/.local/state/nvim/undodir/` holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664. Shredded. `make secrets-edit` now hardens `$EDITOR` before handing it plaintext, so it cannot recur. Never committed and never left the host, so those three communities were not rotated on that basis. | | Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. | | Pre-purge objects still served by GitHub after the history rewrite | The 2026-08-19 rewrite (`021d2b6`) removed both secrets above from every *reachable* commit, but GitHub still serves the orphaned objects by SHA. Verified 2026-08-26: `647d90a`, `21afcad`, `efb2632` and `ee3d443` all still resolve through the API, and the tree at `21afcad` still lists `certificates/Gandalf.Gondor.Lab/ca-key.pem` and `cert-key.pem`. Garbage collection requested from GitHub Support on 2026-08-26 — **pending**; this is the [purge runbook](docs/runbooks/purge-git-history.md)'s *Afterwards* step, and it is the last one outstanding. The repository has no forks and a network count of 0, so nothing else is perpetuating them. Both credentials were rotated *before* the rewrite, so this changes nothing about their status: the old keys and the old community remain superseded and must still be treated as public. Re-check with `gh api repos/Gerrrt/HomeLab/commits/647d90a --jq .sha` — a `404` means GitHub has collected them. | +| Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. | +| Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control: only Hicks (50) and Winterfell (99) can reach the host at all, and never an untrusted segment. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is diff --git a/docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md b/docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md new file mode 100644 index 0000000..3e8b428 --- /dev/null +++ b/docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md @@ -0,0 +1,90 @@ +# ADR-0012: Publish only the ports something off-host uses + +**Status:** Accepted · 2026-08 + +## Context + +The observability stack published five ports on `${BIND_ADDR:-0.0.0.0}` because +one variable governed all of them. Only Grafana authenticates. The others each +carry a write surface: Prometheus runs with `--web.enable-remote-write-receiver` +and `--web.enable-lifecycle`, Loki runs `auth_enabled: false` with a delete API, +and Alertmanager lets any caller create a silence. + +The last of those is the one that prompted this. Silencing an alert switches off +monitoring, and the record of the act lives in the system being switched off. It +is the only one of the three whose abuse is designed to leave no trace. + +[ADR-0002](0002-vlan-segmentation-strategy.md)'s default-deny means only Hicks +(50) can reach Winterfell (99), so none of this was ever exposed to an untrusted +segment. But it made one firewall rule the entire control, and ADR-0002 already +records the gap that leaves: *"A compromised workstation reaches Winterfell."* + +The question was whether the wide default was earning anything. Checking rather +than assuming turned out to matter, and in both directions: + +- `docs/roadmap.md` said only the monitoring host ran an agent. It was a day out + of date. `oracle` had been remote-writing to `10.0.99.20:9090` and pushing to + `:3100` for about eighteen hours — 378 distinct metric names, and it is not a + scrape target, so those published ports are its only path. Prometheus and Loki + are load-bearing. +- Alertmanager had no off-host client at all, and still does not. Grafana + proxies it over the compose network through an authenticated datasource, so + every operator path to a silence already avoided port 9093. + +A second assumption did not survive either: the monitoring host is a 2012 +MacBook Pro, so narrowing `BIND_ADDR` from `0.0.0.0` to the management IP looked +like cheap defence in depth. It is not — the machine has one interface, a USB +ethernet adapter already on VLAN 99. `0.0.0.0` and `10.0.99.20` are the same +thing here, and pinning the literal address would only add a way for the stack +to fail if it ever moved. + +## Decision + +A port is published to a host interface when something off the host uses it, and +not otherwise. The test is a named client, not a plausible future one. + +- **Grafana (3000)** — browsers on Hicks. Published, and the only published + service that authenticates. +- **Prometheus (9090)** and **Loki (3100)** — `oracle`'s Alloy agent. + Published. +- **Alloy syslog (1514/udp)** — `morpheus`. Published; pfSense cannot reach + loopback. +- **Alertmanager (9093)** — nothing. Bound to `127.0.0.1`, hard-coded beside the + Alloy debug UI (12345) that was already there. + +Loopback is hard-coded rather than made a variable. `.env` is regenerated from +`.env.example` on every `make up`, so a variable would advertise a runtime knob +the deploy path does not provide; and publishing one of these hands a VLAN write +access to a store, which should arrive as a reviewed diff against the pull +request template's *"No new port published to a VLAN that could not already +reach the service"* checkbox. + +No service flags change. `--web.enable-remote-write-receiver` is what `oracle` +uses, `--web.enable-lifecycle` is what `scripts/reload-config.sh` uses from +inside the container, and Loki's `auth_enabled: false` is correct for one +tenant. The flags were never the problem; reachability was. + +## Consequences + +- **The silence surface is closed.** The one write path whose abuse leaves no + trace is no longer reachable from any VLAN. +- **Prometheus and Loki remain an accepted residual, not a fixed one.** They are + published, unauthenticated, and writable by anything that can route to + `10.0.99.20` — which today means Hicks and Winterfell. The firewall is still + the only control there. Recorded in [`SECURITY.md`](../../SECURITY.md) rather + than quietly carried; closing it needs authentication in front of the ingest + ports, which is a separate piece of work and a new secret to rotate. +- Alertmanager's `--web.external-url` still points at `10.0.99.20:9093`, so the + `externalURL` field in webhook payloads now only opens on the monitoring host + or through an SSH tunnel. No receiver renders it, so no notification text + changes. Off-host, silences are in Grafana under Alerting; on-host, + `docker exec alertmanager amtool` still works. +- `127.0.0.1` is a host boundary, not a user one. Any local account on the + monitoring host, and any container on the host network, still reaches + Alertmanager unauthenticated. +- **The rule generalises to the next agent.** Deploying Alloy to `Saruman` + ([#88](https://github.com/Gerrrt/HomeLab/issues/88)) needs no bind change, + because `oracle` already made Prometheus and Loki published services. Adding a + service with no off-host client should default to loopback. +- Supersedes nothing. It fills in a layer ADR-0002's consequences section + admitted was missing, at a scale where a bastion is still not worth building. diff --git a/docs/architecture.md b/docs/architecture.md index 0cf6f25..1e2e537 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -175,19 +175,35 @@ re-shard of everything. Reasoning in | Service | Port | Bound to | Notes | | --- | --- | --- | --- | -| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human | -| Prometheus | 9090 | `${BIND_ADDR}` | Also the remote-write receiver for agents | -| Loki | 3100 | `${BIND_ADDR}` | Push endpoint for agents | -| Alertmanager | 9093 | `${BIND_ADDR}` | | +| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only published service that authenticates | +| Prometheus | 9090 | `${BIND_ADDR}` | Remote-write receiver — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | +| Loki | 3100 | `${BIND_ADDR}` | Push endpoint — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | +| Alertmanager | 9093 | `127.0.0.1` | Nothing off-host uses it; silences are reached through Grafana | | Alloy | 12345 | `127.0.0.1` | Debug UI, deliberately not exposed | | Alloy syslog | 1514/udp | `${BIND_ADDR}` | Network syslog receiver — pfSense pushes here | | snmp-exporter | 9116 | *compose network only* | Never published to a host interface | | blackbox-exporter | 9115 | *compose network only* | Never published — an open prober is an SSRF primitive | -`BIND_ADDR` defaults to `0.0.0.0` and is set in `.env`. Setting it to the host's -VLAN 99 address confines the whole stack to the management segment; the -published ports exist because agents on other hosts need to reach Prometheus and -Loki. +A port is published only when something off this host uses it +([ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md)). Grafana +is opened in a browser from Hicks, the syslog receiver takes pushes from +`morpheus`, and Prometheus and Loki take metrics and logs from `oracle`'s Alloy +agent. Alertmanager has no such client, so it binds to `127.0.0.1`; silences are +reached through Grafana, which proxies it over the compose network behind a +login. + +`BIND_ADDR` governs the four that are published. It defaults to `0.0.0.0` and is +set in `stacks/observability/.env.example` — `.env` is regenerated from that +file on every `make up`, so the committed value is the deployed one. Setting it +to the host's VLAN 99 address would confine them to the management segment, +which today changes nothing: the host has one interface and it is already on +VLAN 99. + +Prometheus and Loki are published and unauthenticated, which is a real residual +rather than a solved problem — anything that can reach them can read every +metric and log line, inject metrics, and delete log ranges. Default-deny between +VLANs is the only control on that, and it is recorded as such in +[`SECURITY.md`](../SECURITY.md). ## Reference diagrams diff --git a/docs/roadmap.md b/docs/roadmap.md index e42ce36..d31c960 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -32,9 +32,10 @@ issues intact. Nothing was summarised away. seconds at gigabit. Now actionable — the switch has been polling since the faults in [#22](https://github.com/Gerrrt/HomeLab/issues/22) cleared. The walk has a trap in it; the issue carries the detail. -- **[#88](https://github.com/Gerrrt/HomeLab/issues/88) Deploy Alloy to `Saruman` - and `oracle`.** Only the monitoring host runs an agent; `morpheus` reaches Loki - by network syslog and has no host metrics. +- **[#88](https://github.com/Gerrrt/HomeLab/issues/88) Deploy Alloy to + `Saruman`.** `oracle` has had an agent since 2026-08-30 and is remote-writing + host metrics and pushing logs; `Saruman` is the one left. `morpheus` reaches + Loki by network syslog and has no host metrics. - **[#89](https://github.com/Gerrrt/HomeLab/issues/89) Extend Suricata to Degens (VLAN 10).** One interface at a time, once Skids has been quiet and understood for a few days. → [runbook](runbooks/enable-suricata.md) diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 10f0aa5..bba9779 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -54,6 +54,15 @@ curl -s localhost:9093/-/healthy # Alertmanager curl -sk https://localhost:3000/api/health # Grafana (-k: lab CA) ``` +The `curl`s pass wherever the services are bound, so confirm the binds +themselves are what `docs/architecture.md` claims (#70): + +```bash +ss -ltn | grep -E ':(9093|12345)' # 127.0.0.1 — Alertmanager and Alloy +ss -ltn | grep -E ':(9090|3100|3000)' # BIND_ADDR — 0.0.0.0 by default +ss -lun | grep ':1514' # BIND_ADDR +``` + Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index b4e1f4f..eae28f5 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -344,6 +344,10 @@ devices. It also means published ports do not route, so reach the services with `docker exec` rather than from the host — and it is why `GF_INSTALL_PLUGINS` has to be emptied above. +`BIND_ADDR=127.0.0.1` in the export below is belt-and-braces on top of that. It +covers the rehearsal Prometheus, Loki, Grafana and syslog receiver; Alertmanager +is pinned to loopback in `compose.yaml` and ignores it (#70). + ```bash export COMPOSE_PROJECT_NAME=rehearse BIND_ADDR=127.0.0.1 \ PROMETHEUS_PORT=19090 ALERTMANAGER_PORT=19093 LOKI_PORT=13100 \ diff --git a/docs/security.md b/docs/security.md index 04b6d4f..2b52868 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,6 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs | +| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | What it explicitly does **not** defend against: a determined attacker with @@ -67,6 +68,19 @@ one that admits the exception. Everything else — IoT, media, guest — gets internet and nothing more. +Segmentation is doing more work here than it should have to. A workstation on +Hicks that can reach `10.0.99.20` can write to the metric and log stores without +a credential, because Prometheus and Loki publish unauthenticated ingest ports +for `oracle`'s agent to use — which is exactly the failure ADR-0002 predicted +when it recorded that "a compromised workstation reaches Winterfell". That is an +accepted residual, recorded in [`SECURITY.md`](../SECURITY.md), not a solved +problem. + +What has been taken off the firewall's shoulders is Alertmanager. It had no +off-host client, so it now binds to `127.0.0.1` and reaching VLAN 99 no longer +lets anyone silence an alert; see +[ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). + The IoT segment is the one that justifies the whole exercise. It holds cameras, a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya white-noise machine. Every one of those is a network-connected computer running @@ -221,6 +235,12 @@ complete state table and interface topology. They are credentials. its own unprivileged UID. - `snmp-exporter` is never published to a host interface — it is reachable only on the compose network. +- Alertmanager binds to `127.0.0.1` only. It is unauthenticated, and a silence + is how monitoring gets switched off — quietly, since the record lives in the + system being switched off. Nothing off-host used the port; silences are + reached through Grafana. Prometheus and Loki are *not* in this list: they stay + published for `oracle`'s agent and remain an accepted residual. See + [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). - The Alloy debug UI binds to `127.0.0.1` only. - The Docker socket is mounted read-only into Alloy. - All images are pinned to explicit versions, so an upstream compromise cannot diff --git a/stacks/observability/.env.example b/stacks/observability/.env.example index 68085fe..a1ee1e5 100644 --- a/stacks/observability/.env.example +++ b/stacks/observability/.env.example @@ -1,19 +1,30 @@ # Non-sensitive tunables for the observability stack. # -# Copy to .env and edit, or just rely on the defaults baked into compose.yaml. +# Edit this file, not .env — scripts/render-config.sh regenerates .env from it +# on every `make up`. Or rely on the defaults baked into compose.yaml. # Secrets do NOT belong here — see secrets/README.md. # # scripts/render-config.sh appends the decrypted secret values to .env at # deploy time, which is why .env itself is gitignored. -# Address the published ports bind to. 0.0.0.0 exposes the stack to every VLAN -# the host can reach; set this to the management IP to restrict it to VLAN 99. +# Address the published ports bind to. 0.0.0.0 exposes them to every VLAN the +# host can reach; set this to the management IP to restrict them to VLAN 99. +# +# Change it *here*, not in .env: scripts/render-config.sh rewrites .env from +# this file on every `make up`, so an edit to .env lasts until the next deploy +# and no longer. The committed value is the deployed value. +# +# Alertmanager is not governed by this — it is pinned to 127.0.0.1 in +# compose.yaml, being the one service with no off-host client (#70). BIND_ADDR=0.0.0.0 -# Used for Alertmanager's external URL in notification links. +# Used for Alertmanager's external URL in notification links. Since 9093 binds +# to loopback (#70) that link only opens on this host; off-host, silences are in +# Grafana under Alerting. MONITORING_HOST=10.0.99.20 -# Published ports. +# Host ports. ALERTMANAGER and ALLOY are published on 127.0.0.1; the rest on +# BIND_ADDR. See the note above. PROMETHEUS_PORT=9090 ALERTMANAGER_PORT=9093 LOKI_PORT=3100 diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 8891218..05e401d 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -9,17 +9,25 @@ make up # from the repository root | Service | Image | Port | Purpose | | --- | --- | --- | --- | | `prometheus` | `prom/prometheus` | 9090 | Metrics store, remote-write receiver, rule evaluation | -| `alertmanager` | `prom/alertmanager` | 9093 | Alert routing, grouping, inhibition | +| `alertmanager` | `prom/alertmanager` | 9093 (localhost) | Alert routing, grouping, inhibition | | `loki` | `grafana/loki` | 3100 | Log store | -| `grafana` | `grafana/grafana-oss` | 3000 | Dashboards | +| `grafana` | `grafana/grafana-oss` | 3000 | Dashboards — the only published UI, and the only one that authenticates | | `snmp-exporter` | `prom/snmp-exporter` | *internal* | SNMP polling proxy | | `alloy` | `grafana/alloy` | 12345 (localhost) | Metric and log collection | +"(localhost)" means bound to `127.0.0.1`: reachable from the monitoring host +itself and over the compose network, and from no VLAN at all. A port is +published only when something off-host uses it, and nothing off-host uses +Alertmanager (#70). The rest, plus the Alloy syslog receiver on 1514/udp, bind +to `BIND_ADDR`. Reasoning in +[`docs/architecture.md`](../../docs/architecture.md#ports). + ## Layout ```text compose.yaml all seven services, one network, health-gated ordering .env.example non-sensitive tunables (ports, retention, bind address) + edit this, not .env — .env is regenerated on `make up` prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index bc52b71..125217e 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -46,6 +46,11 @@ services: - ./prometheus/targets:/etc/prometheus/targets:ro - prometheus-data:/prometheus ports: + # Published, and unauthenticated with it: anything that can reach this + # port can read every metric and inject more through the remote-write + # receiver. It stays published because `oracle` remote-writes here — that + # is a real client, not a hypothetical one (#70). Recorded as an accepted + # residual in SECURITY.md rather than left implicit. - "${BIND_ADDR:-0.0.0.0}:${PROMETHEUS_PORT:-9090}:9090" healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"] @@ -70,6 +75,10 @@ services: command: - --config.file=/etc/alertmanager/alertmanager.yaml - --storage.path=/alertmanager + # Seeds the externalURL field in webhook payloads and nothing else — no + # receiver here renders it. Since #70 that link opens only on the + # monitoring host. Left pointing at the real address rather than + # 127.0.0.1, because that is still the host you would tunnel to. - --web.external-url=http://${MONITORING_HOST:-10.0.99.20}:${ALERTMANAGER_PORT:-9093} volumes: - ./alertmanager/alertmanager.yaml:/etc/alertmanager/alertmanager.yaml:ro @@ -79,7 +88,13 @@ services: - ./alertmanager/.rendered:/etc/alertmanager/secrets:ro - alertmanager-data:/alertmanager ports: - - "${BIND_ADDR:-0.0.0.0}:${ALERTMANAGER_PORT:-9093}:9093" + # Loopback only, and the only one of the four that can be (#70). Anyone + # who can reach an unauthenticated Alertmanager can create a silence, + # which switches off an alert using the same system that would otherwise + # have recorded the act. Unlike Prometheus and Loki above, nothing + # off-host talks to it: silences are reached through Grafana, which + # proxies this service over the compose network behind a login. + - "127.0.0.1:${ALERTMANAGER_PORT:-9093}:9093" healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:9093/-/healthy"] interval: 30s @@ -105,6 +120,9 @@ services: - ./loki/rules:/etc/loki/rules/fake:ro - loki-data:/loki ports: + # Published for the same reason as Prometheus above, and with the same + # residual: auth_enabled is false, so the push and delete APIs are open to + # whatever can reach the port. `oracle`'s Alloy pushes here (#70). - "${BIND_ADDR:-0.0.0.0}:${LOKI_PORT:-3100}:3100" # No healthcheck on loki, deliberately. @@ -265,6 +283,9 @@ services: group_add: - "${RENDER_GID:?run make render}" ports: + # The only published port a human is meant to open, and the only one that + # authenticates: anonymous access and sign-up are both off, and the admin + # password comes from SOPS. - "${BIND_ADDR:-0.0.0.0}:${GRAFANA_PORT:-3000}:3000" depends_on: prometheus: