fix(observability): bind Alertmanager to loopback (#70)

README.md

@@ -165,7 +165,7 @@ rack; a dashed border means egress only. Full topology and data flow in
 ├── docs/
 │   ├── architecture.md  network.md  hardware.md
 │   ├── observability.md  security.md  roadmap.md
-│   ├── adr/                  # 11 architecture decision records
+│   ├── adr/                  # 12 architecture decision records
 │   └── runbooks/             # deploy, add device, rotate creds, certs, key backup,
 │                             #   purge, restore the firewall, restore the stack,
 │                             #   ship firewall logs, verify the alert path,
@@ -195,7 +195,9 @@ Prometheus verifies it with the CA, so `make up` renders nothing until they
 exist. Details in
 [`docs/runbooks/generate-certificates.md`](docs/runbooks/generate-certificates.md).
 
-Grafana on `:3000` over https, Prometheus on `:9090`. Grafana's certificate is
+Grafana on `:3000` over https, Prometheus on `:9090`. Alertmanager binds to
+`127.0.0.1` and is reached through Grafana
+([#70](https://github.com/Gerrrt/HomeLab/issues/70)). Grafana's certificate is
 signed by the lab's own CA, so a browser warns and `curl` needs `-k` until you
 trust `certificates/ca.pem` — step 4 of that runbook. Full procedure,
 verification steps and troubleshooting in

SECURITY.md

@@ -44,6 +44,8 @@ is a very different thing from an overlooked one. Full detail in
 | Decrypted secrets in editor undo files, written by `make secrets-edit` | Found 2026-08-20: three files under `~/.local/state/nvim/undodir/` holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664. Shredded. `make secrets-edit` now hardens `$EDITOR` before handing it plaintext, so it cannot recur. Never committed and never left the host, so those three communities were not rotated on that basis. |
 | Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. |
 | Pre-purge objects still served by GitHub after the history rewrite | The 2026-08-19 rewrite (`021d2b6`) removed both secrets above from every *reachable* commit, but GitHub still serves the orphaned objects by SHA. Verified 2026-08-26: `647d90a`, `21afcad`, `efb2632` and `ee3d443` all still resolve through the API, and the tree at `21afcad` still lists `certificates/Gandalf.Gondor.Lab/ca-key.pem` and `cert-key.pem`. Garbage collection requested from GitHub Support on 2026-08-26 — **pending**; this is the [purge runbook](docs/runbooks/purge-git-history.md)'s *Afterwards* step, and it is the last one outstanding. The repository has no forks and a network count of 0, so nothing else is perpetuating them. Both credentials were rotated *before* the rewrite, so this changes nothing about their status: the old keys and the old community remain superseded and must still be treated as public. Re-check with `gh api repos/Gerrrt/HomeLab/commits/647d90a --jq .sha` — a `404` means GitHub has collected them. |
+| Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. |
+| Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control: only Hicks (50) and Winterfell (99) can reach the host at all, and never an untrusted segment. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. |
 | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. |
 
 The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is

docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md

@@ -0,0 +1,90 @@
+# ADR-0012: Publish only the ports something off-host uses
+
+**Status:** Accepted · 2026-08
+
+## Context
+
+The observability stack published five ports on `${BIND_ADDR:-0.0.0.0}` because
+one variable governed all of them. Only Grafana authenticates. The others each
+carry a write surface: Prometheus runs with `--web.enable-remote-write-receiver`
+and `--web.enable-lifecycle`, Loki runs `auth_enabled: false` with a delete API,
+and Alertmanager lets any caller create a silence.
+
+The last of those is the one that prompted this. Silencing an alert switches off
+monitoring, and the record of the act lives in the system being switched off. It
+is the only one of the three whose abuse is designed to leave no trace.
+
+[ADR-0002](0002-vlan-segmentation-strategy.md)'s default-deny means only Hicks
+(50) can reach Winterfell (99), so none of this was ever exposed to an untrusted
+segment. But it made one firewall rule the entire control, and ADR-0002 already
+records the gap that leaves: *"A compromised workstation reaches Winterfell."*
+
+The question was whether the wide default was earning anything. Checking rather
+than assuming turned out to matter, and in both directions:
+
+- `docs/roadmap.md` said only the monitoring host ran an agent. It was a day out
+  of date. `oracle` had been remote-writing to `10.0.99.20:9090` and pushing to
+  `:3100` for about eighteen hours — 378 distinct metric names, and it is not a
+  scrape target, so those published ports are its only path. Prometheus and Loki
+  are load-bearing.
+- Alertmanager had no off-host client at all, and still does not. Grafana
+  proxies it over the compose network through an authenticated datasource, so
+  every operator path to a silence already avoided port 9093.
+
+A second assumption did not survive either: the monitoring host is a 2012
+MacBook Pro, so narrowing `BIND_ADDR` from `0.0.0.0` to the management IP looked
+like cheap defence in depth. It is not — the machine has one interface, a USB
+ethernet adapter already on VLAN 99. `0.0.0.0` and `10.0.99.20` are the same
+thing here, and pinning the literal address would only add a way for the stack
+to fail if it ever moved.
+
+## Decision
+
+A port is published to a host interface when something off the host uses it, and
+not otherwise. The test is a named client, not a plausible future one.
+
+- **Grafana (3000)** — browsers on Hicks. Published, and the only published
+  service that authenticates.
+- **Prometheus (9090)** and **Loki (3100)** — `oracle`'s Alloy agent.
+  Published.
+- **Alloy syslog (1514/udp)** — `morpheus`. Published; pfSense cannot reach
+  loopback.
+- **Alertmanager (9093)** — nothing. Bound to `127.0.0.1`, hard-coded beside the
+  Alloy debug UI (12345) that was already there.
+
+Loopback is hard-coded rather than made a variable. `.env` is regenerated from
+`.env.example` on every `make up`, so a variable would advertise a runtime knob
+the deploy path does not provide; and publishing one of these hands a VLAN write
+access to a store, which should arrive as a reviewed diff against the pull
+request template's *"No new port published to a VLAN that could not already
+reach the service"* checkbox.
+
+No service flags change. `--web.enable-remote-write-receiver` is what `oracle`
+uses, `--web.enable-lifecycle` is what `scripts/reload-config.sh` uses from
+inside the container, and Loki's `auth_enabled: false` is correct for one
+tenant. The flags were never the problem; reachability was.
+
+## Consequences
+
+- **The silence surface is closed.** The one write path whose abuse leaves no
+  trace is no longer reachable from any VLAN.
+- **Prometheus and Loki remain an accepted residual, not a fixed one.** They are
+  published, unauthenticated, and writable by anything that can route to
+  `10.0.99.20` — which today means Hicks and Winterfell. The firewall is still
+  the only control there. Recorded in [`SECURITY.md`](../../SECURITY.md) rather
+  than quietly carried; closing it needs authentication in front of the ingest
+  ports, which is a separate piece of work and a new secret to rotate.
+- Alertmanager's `--web.external-url` still points at `10.0.99.20:9093`, so the
+  `externalURL` field in webhook payloads now only opens on the monitoring host
+  or through an SSH tunnel. No receiver renders it, so no notification text
+  changes. Off-host, silences are in Grafana under Alerting; on-host,
+  `docker exec alertmanager amtool` still works.
+- `127.0.0.1` is a host boundary, not a user one. Any local account on the
+  monitoring host, and any container on the host network, still reaches
+  Alertmanager unauthenticated.
+- **The rule generalises to the next agent.** Deploying Alloy to `Saruman`
+  ([#88](https://github.com/Gerrrt/HomeLab/issues/88)) needs no bind change,
+  because `oracle` already made Prometheus and Loki published services. Adding a
+  service with no off-host client should default to loopback.
+- Supersedes nothing. It fills in a layer ADR-0002's consequences section
+  admitted was missing, at a scale where a bastion is still not worth building.

docs/architecture.md

@@ -175,19 +175,35 @@ re-shard of everything. Reasoning in
 
 | Service | Port | Bound to | Notes |
 | --- | --- | --- | --- |
-| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human |
-| Prometheus | 9090 | `${BIND_ADDR}` | Also the remote-write receiver for agents |
-| Loki | 3100 | `${BIND_ADDR}` | Push endpoint for agents |
-| Alertmanager | 9093 | `${BIND_ADDR}` | |
+| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only published service that authenticates |
+| Prometheus | 9090 | `${BIND_ADDR}` | Remote-write receiver — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) |
+| Loki | 3100 | `${BIND_ADDR}` | Push endpoint — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) |
+| Alertmanager | 9093 | `127.0.0.1` | Nothing off-host uses it; silences are reached through Grafana |
 | Alloy | 12345 | `127.0.0.1` | Debug UI, deliberately not exposed |
 | Alloy syslog | 1514/udp | `${BIND_ADDR}` | Network syslog receiver — pfSense pushes here |
 | snmp-exporter | 9116 | *compose network only* | Never published to a host interface |
 | blackbox-exporter | 9115 | *compose network only* | Never published — an open prober is an SSRF primitive |
 
-`BIND_ADDR` defaults to `0.0.0.0` and is set in `.env`. Setting it to the host's
-VLAN 99 address confines the whole stack to the management segment; the
-published ports exist because agents on other hosts need to reach Prometheus and
-Loki.
+A port is published only when something off this host uses it
+([ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md)). Grafana
+is opened in a browser from Hicks, the syslog receiver takes pushes from
+`morpheus`, and Prometheus and Loki take metrics and logs from `oracle`'s Alloy
+agent. Alertmanager has no such client, so it binds to `127.0.0.1`; silences are
+reached through Grafana, which proxies it over the compose network behind a
+login.
+
+`BIND_ADDR` governs the four that are published. It defaults to `0.0.0.0` and is
+set in `stacks/observability/.env.example` — `.env` is regenerated from that
+file on every `make up`, so the committed value is the deployed one. Setting it
+to the host's VLAN 99 address would confine them to the management segment,
+which today changes nothing: the host has one interface and it is already on
+VLAN 99.
+
+Prometheus and Loki are published and unauthenticated, which is a real residual
+rather than a solved problem — anything that can reach them can read every
+metric and log line, inject metrics, and delete log ranges. Default-deny between
+VLANs is the only control on that, and it is recorded as such in
+[`SECURITY.md`](../SECURITY.md).
 
 ## Reference diagrams
 

docs/roadmap.md

@@ -32,9 +32,10 @@ issues intact. Nothing was summarised away.
   seconds at gigabit. Now actionable — the switch has been polling since the
   faults in [#22](https://github.com/Gerrrt/HomeLab/issues/22) cleared. The walk
   has a trap in it; the issue carries the detail.
-- **[#88](https://github.com/Gerrrt/HomeLab/issues/88) Deploy Alloy to `Saruman`
-  and `oracle`.** Only the monitoring host runs an agent; `morpheus` reaches Loki
-  by network syslog and has no host metrics.
+- **[#88](https://github.com/Gerrrt/HomeLab/issues/88) Deploy Alloy to
+  `Saruman`.** `oracle` has had an agent since 2026-08-30 and is remote-writing
+  host metrics and pushing logs; `Saruman` is the one left. `morpheus` reaches
+  Loki by network syslog and has no host metrics.
 - **[#89](https://github.com/Gerrrt/HomeLab/issues/89) Extend Suricata to Degens
   (VLAN 10).** One interface at a time, once Skids has been quiet and understood
   for a few days. → [runbook](runbooks/enable-suricata.md)

docs/runbooks/deploy-stack.md

@@ -54,6 +54,15 @@ curl -s localhost:9093/-/healthy            # Alertmanager
 curl -sk https://localhost:3000/api/health  # Grafana (-k: lab CA)
 ```
 
+The `curl`s pass wherever the services are bound, so confirm the binds
+themselves are what `docs/architecture.md` claims (#70):
+
+```bash
+ss -ltn | grep -E ':(9093|12345)'        # 127.0.0.1 — Alertmanager and Alloy
+ss -ltn | grep -E ':(9090|3100|3000)'    # BIND_ADDR — 0.0.0.0 by default
+ss -lun | grep ':1514'                   # BIND_ADDR
+```
+
 Then in the UI:
 
 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets

docs/runbooks/restore-the-stack.md

@@ -344,6 +344,10 @@ devices. It also means published ports do not route, so reach the services with
 `docker exec` rather than from the host — and it is why `GF_INSTALL_PLUGINS`
 has to be emptied above.
 
+`BIND_ADDR=127.0.0.1` in the export below is belt-and-braces on top of that. It
+covers the rehearsal Prometheus, Loki, Grafana and syslog receiver; Alertmanager
+is pinned to loopback in `compose.yaml` and ignores it (#70).
+
 ```bash
 export COMPOSE_PROJECT_NAME=rehearse BIND_ADDR=127.0.0.1 \
   PROMETHEUS_PORT=19090 ALERTMANAGER_PORT=19093 LOKI_PORT=13100 \

docs/security.md

@@ -15,6 +15,7 @@ What this network is actually built to survive:
 | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
 | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
 | Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs |
+| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
 | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below |
 
 What it explicitly does **not** defend against: a determined attacker with
@@ -67,6 +68,19 @@ one that admits the exception.
 
 Everything else — IoT, media, guest — gets internet and nothing more.
 
+Segmentation is doing more work here than it should have to. A workstation on
+Hicks that can reach `10.0.99.20` can write to the metric and log stores without
+a credential, because Prometheus and Loki publish unauthenticated ingest ports
+for `oracle`'s agent to use — which is exactly the failure ADR-0002 predicted
+when it recorded that "a compromised workstation reaches Winterfell". That is an
+accepted residual, recorded in [`SECURITY.md`](../SECURITY.md), not a solved
+problem.
+
+What has been taken off the firewall's shoulders is Alertmanager. It had no
+off-host client, so it now binds to `127.0.0.1` and reaching VLAN 99 no longer
+lets anyone silence an alert; see
+[ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md).
+
 The IoT segment is the one that justifies the whole exercise. It holds cameras,
 a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya
 white-noise machine. Every one of those is a network-connected computer running
@@ -221,6 +235,12 @@ complete state table and interface topology. They are credentials.
   its own unprivileged UID.
 - `snmp-exporter` is never published to a host interface — it is reachable only
   on the compose network.
+- Alertmanager binds to `127.0.0.1` only. It is unauthenticated, and a silence
+  is how monitoring gets switched off — quietly, since the record lives in the
+  system being switched off. Nothing off-host used the port; silences are
+  reached through Grafana. Prometheus and Loki are *not* in this list: they stay
+  published for `oracle`'s agent and remain an accepted residual. See
+  [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md).
 - The Alloy debug UI binds to `127.0.0.1` only.
 - The Docker socket is mounted read-only into Alloy.
 - All images are pinned to explicit versions, so an upstream compromise cannot

stacks/observability/.env.example

@@ -1,19 +1,30 @@
 # Non-sensitive tunables for the observability stack.
 #
-# Copy to .env and edit, or just rely on the defaults baked into compose.yaml.
+# Edit this file, not .env — scripts/render-config.sh regenerates .env from it
+# on every `make up`. Or rely on the defaults baked into compose.yaml.
 # Secrets do NOT belong here — see secrets/README.md.
 #
 # scripts/render-config.sh appends the decrypted secret values to .env at
 # deploy time, which is why .env itself is gitignored.
 
-# Address the published ports bind to. 0.0.0.0 exposes the stack to every VLAN
-# the host can reach; set this to the management IP to restrict it to VLAN 99.
+# Address the published ports bind to. 0.0.0.0 exposes them to every VLAN the
+# host can reach; set this to the management IP to restrict them to VLAN 99.
+#
+# Change it *here*, not in .env: scripts/render-config.sh rewrites .env from
+# this file on every `make up`, so an edit to .env lasts until the next deploy
+# and no longer. The committed value is the deployed value.
+#
+# Alertmanager is not governed by this — it is pinned to 127.0.0.1 in
+# compose.yaml, being the one service with no off-host client (#70).
 BIND_ADDR=0.0.0.0
 
-# Used for Alertmanager's external URL in notification links.
+# Used for Alertmanager's external URL in notification links. Since 9093 binds
+# to loopback (#70) that link only opens on this host; off-host, silences are in
+# Grafana under Alerting.
 MONITORING_HOST=10.0.99.20
 
-# Published ports.
+# Host ports. ALERTMANAGER and ALLOY are published on 127.0.0.1; the rest on
+# BIND_ADDR. See the note above.
 PROMETHEUS_PORT=9090
 ALERTMANAGER_PORT=9093
 LOKI_PORT=3100