From d3a69b4b85440c6097636be471b5942b474ce9b0 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 31 Aug 2026 05:02:23 +0000 Subject: [PATCH 1/2] docs(tls): say Grafana terminates TLS, and issue a leaf that verifies (#74) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Grafana has served https since the roadmap entry that recorded it, but three places still said nothing in the stack terminated TLS. Two of the five the issue named had already been fixed; the rest are here. The worst was not a comment. gen-certs.sh renders its own header for --help via `sed -n '2,32p'`, so `make certs ARGS=--help` printed "Nothing in this repository terminates TLS yet" to the operator running the step that exists precisely because Grafana does. That range had also drifted and was truncating the last paragraph mid-sentence, so --help now prints the comment block by where it ends rather than by line number. The runbook pointed at roadmap.md for "putting the rest behind TLS", which nothing tracks — #182 is authentication of the ingest ports, #91 is TLS-expiry probing. Prometheus, Alertmanager and Loki really are plain HTTP, so that half stays; the pointer now goes to security.md, where it is already recorded as an accepted residual with firewall default-deny as the control. Found in the same pass, and worse than stale wording: every documented `make certs` leaf command omitted `--dns grafana`, while prometheus.yaml sets `server_name: grafana` and verifies that short name over the compose network. The running cert carries the SAN, so nothing was broken here — but a fresh clone following the docs would mint a leaf without it and the grafana scrape would fail x509 name verification, for a reason that reads like a trust problem. All seven invocations now pass it, with the reason written down next to the `--ip` note. Also gives Grafana its scheme in the two places a reader starts: the architecture diagram and ports table, and the stack's service table. Verified: the documented command now reproduces the live certificate's SANs exactly (DNS:grafana.matrix.elysium, DNS:grafana, IP:10.0.99.20); `make validate` passes; up{job="grafana"} is still 1 over https. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- docs/architecture.md | 4 ++-- docs/runbooks/deploy-stack.md | 2 +- docs/runbooks/generate-certificates.md | 22 ++++++++++++++++------ scripts/gen-certs.sh | 18 ++++++++++++------ scripts/render-config.sh | 2 +- stacks/observability/README.md | 2 +- 7 files changed, 34 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index b0de7ce..d21bde4 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,7 @@ git clone https://github.com/Gerrrt/HomeLab.git && cd HomeLab make secrets-init # generate an age keypair, create the encrypted secrets file make secrets-edit # fill in real values make certs ARGS=--ca # create the lab CA -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" # Grafana's leaf +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" # Grafana's leaf make validate # everything CI runs make up # render config and start the stack ``` diff --git a/docs/architecture.md b/docs/architecture.md index 1e2e537..3229e8d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -116,7 +116,7 @@ graph LR PROM[("Prometheus
:9090 · 30d")] LOKI[("Loki
:3100 · 30d")] AM["Alertmanager
:9093"] - GRAF["Grafana
:3000"] + GRAF["Grafana
:3000 · https"] end OUT([Webhook
notification]) @@ -175,7 +175,7 @@ re-shard of everything. Reasoning in | Service | Port | Bound to | Notes | | --- | --- | --- | --- | -| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only published service that authenticates | +| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only service that terminates TLS or authenticates — `https://`, on a lab-CA certificate a browser will warn about until you trust `certificates/ca.pem` | | Prometheus | 9090 | `${BIND_ADDR}` | Remote-write receiver — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | | Loki | 3100 | `${BIND_ADDR}` | Push endpoint — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | | Alertmanager | 9093 | `127.0.0.1` | Nothing off-host uses it; silences are reached through Grafana | diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 5f9792d..bfc6196 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -23,7 +23,7 @@ make secrets-edit # replace every change-me value # TLS. Grafana serves https from this leaf and Prometheus verifies it with the # CA — see generate-certificates.md. Both are required before the stack starts. make certs ARGS=--ca -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" make validate # confirm the configs are sound before starting anything make up diff --git a/docs/runbooks/generate-certificates.md b/docs/runbooks/generate-certificates.md index 0afe918..ba2ffa7 100644 --- a/docs/runbooks/generate-certificates.md +++ b/docs/runbooks/generate-certificates.md @@ -15,9 +15,11 @@ that rewrite as compromised, including the CA. What follows replaces it. > [`deploy-stack.md`](deploy-stack.md), not a someday step. `make render` > refuses to run until they exist, because before it did, `make up` bind-mounted > the absent files and Docker created directories in their place -> ([#69](https://github.com/Gerrrt/HomeLab/issues/69)). Prometheus, Alertmanager -> and Loki still publish plain HTTP on the management VLAN; putting the rest -> behind TLS is tracked in [`roadmap.md`](../roadmap.md). +> ([#69](https://github.com/Gerrrt/HomeLab/issues/69)). Grafana is the only +> service that terminates TLS. Prometheus, Alertmanager and Loki publish plain +> HTTP on the management VLAN, and that is an accepted residual with firewall +> default-deny as the control — not work in progress; see +> [`security.md`](../security.md). ## Where things live @@ -59,7 +61,7 @@ trust store holding it, so that has to be deliberate (`--force`). ## 2. Issue a leaf ```bash -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" ``` Include `--ip` for anything reached by address. `docs/roadmap.md` still lists @@ -67,6 +69,14 @@ internal DNS as unresolved, so in practice most services here are reached by IP, and a certificate without a matching IP SAN will be rejected at the point you most want it to work. +`--dns grafana` is not decoration. Prometheus scrapes the `grafana` job over the +compose network, where the service answers to its short name, so +[`prometheus.yaml`](../../stacks/observability/prometheus/prometheus.yaml) sets +`server_name: grafana` and verifies that name against the leaf. Issue this +certificate with the FQDN alone and Grafana still serves fine in a browser while +the scrape fails with `x509: certificate is valid for grafana.matrix.elysium, +not grafana` — one target down, for a reason that reads like a trust problem. + Two limits the script enforces rather than lets you discover later: - **A certificate with no `subjectAltName` is rejected outright** by every @@ -110,7 +120,7 @@ lab with one certificate is better served by a calendar reminder than by a renewal daemon nobody maintains. ```bash -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --force" +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana --force" ``` Then restart whatever serves it. The CA does not change, so nothing needs @@ -146,7 +156,7 @@ certificate with `-s`, and `-s` is true of a directory: ```console $ ls -ld certificates/grafana.matrix.elysium-key.pem drwxr-xr-x 2 you you 4096 ... certificates/grafana.matrix.elysium-key.pem -$ make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" +$ make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" error: certificates/grafana.matrix.elysium-key.pem already exists — pass --force to replace it ``` diff --git a/scripts/gen-certs.sh b/scripts/gen-certs.sh index 1f78ba0..555373f 100755 --- a/scripts/gen-certs.sh +++ b/scripts/gen-certs.sh @@ -8,10 +8,12 @@ # The control that matters is not the file mode, it is that these never become # tracked files. `make validate` asserts that; so does CI. # -# Nothing in this repository terminates TLS yet. This exists so that when -# something does — Grafana is the first candidate, see docs/roadmap.md — issuing -# a certificate is one command rather than a research project, and so the CA is -# created deliberately rather than improvised at the point of need. +# Grafana terminates TLS with a leaf issued here, and Prometheus verifies that +# certificate against this CA when it scrapes the grafana job — so both are a +# prerequisite of `make up` rather than groundwork for a someday service. What +# this script buys is that issuing one stays a single command rather than a +# research project, and that the CA is created deliberately rather than +# improvised at the point of need. # # Usage: # scripts/gen-certs.sh --ca create the CA (refuses if it exists) @@ -66,7 +68,11 @@ while (($#)); do --dns) DNS+=("${2:?--dns needs a name}"); shift 2 ;; --days) LEAF_DAYS="${2:?--days needs a number}"; shift 2 ;; --force) FORCE=1; shift ;; - -h|--help) sed -n '2,32p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + # Printed straight from the comment block above rather than kept as a + # second copy that drifts. Bounded by where the comments stop, not by a + # line number: the literal 2,32p this replaced had already slipped and was + # truncating the last paragraph mid-sentence. + -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "${BASH_SOURCE[0]}"; exit 0 ;; *) die "unknown argument: $1" ;; esac done @@ -127,7 +133,7 @@ that holds it. Pass --force only if that is what you mean." ok "CA created" printf ' key %s (never commit, never copy off this host)\n' "${CA_KEY#"${REPO_ROOT}"/}" printf ' cert %s (safe to distribute — this is what clients trust)\n' "${CA_CRT#"${REPO_ROOT}"/}" - printf '\nIssue a leaf with:\n scripts/gen-certs.sh --host grafana.matrix.elysium --ip 10.0.99.20\n\n' + printf '\nIssue a leaf with:\n scripts/gen-certs.sh --host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana\n\n' exit 0 fi diff --git a/scripts/render-config.sh b/scripts/render-config.sh index c0e7e4e..6d00ad4 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -93,7 +93,7 @@ Grafana serves https from the leaf and Prometheus verifies it with the CA, so the stack cannot start without them. Generate them with: make certs ARGS=--ca - make certs ARGS=\"--host grafana.matrix.elysium --ip 10.0.99.20\" + make certs ARGS=\"--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana\" Full procedure in docs/runbooks/generate-certificates.md." fi diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 356f7ba..16ea909 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -11,7 +11,7 @@ make up # from the repository root | `prometheus` | `prom/prometheus` | 9090 | Metrics store, remote-write receiver, rule evaluation | | `alertmanager` | `prom/alertmanager` | 9093 (localhost) | Alert routing, grouping, inhibition | | `loki` | `grafana/loki` | 3100 | Log store | -| `grafana` | `grafana/grafana-oss` | 3000 | Dashboards — the only published UI, and the only one that authenticates | +| `grafana` | `grafana/grafana-oss` | 3000 (https) | Dashboards — the only published UI, and the only service that terminates TLS or authenticates | | `snmp-exporter` | `prom/snmp-exporter` | *internal* | SNMP polling proxy | | `alloy` | `grafana/alloy` | 12345 (localhost) | Metric and log collection | From 0cb5cfd8cc950d1ea05f2ec169602caced0b8378 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 31 Aug 2026 05:12:32 +0000 Subject: [PATCH 2/2] docs(tls): the blackbox-exporter this runbook waits for already exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same stale-claim family as the rest of #74, missed on the first pass because I grepped for "terminates TLS" and not for "until that exists". The Renewal section said adding blackbox-exporter for TLS-expiry checks was on roadmap.md, "until that exists". It exists — compose.yaml defines it and prometheus.yaml scrapes it. What does not exist is any expiry checking: it probes only the wiki over plain HTTP, nothing reads probe_ssl_earliest_cert_expiry, and Grafana is not a target, so nothing watches the one certificate this runbook exists to issue. The consequence the paragraph warns about is therefore still true — it was just attributing it to a missing component rather than to unfinished work on a component that is already running. Points at #91, whose roadmap entry is corrected separately. Co-Authored-By: Claude Opus 5 --- docs/runbooks/generate-certificates.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/runbooks/generate-certificates.md b/docs/runbooks/generate-certificates.md index ba2ffa7..59c6c1a 100644 --- a/docs/runbooks/generate-certificates.md +++ b/docs/runbooks/generate-certificates.md @@ -126,9 +126,11 @@ make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana --f Then restart whatever serves it. The CA does not change, so nothing needs re-trusting. -Adding blackbox-exporter for TLS-expiry checks is on -[`roadmap.md`](../roadmap.md); until that exists, expiry is something you find -out about from a browser warning. +blackbox-exporter exists and could carry this, but does not yet: it probes only +the wiki, over plain HTTP, and nothing reads `probe_ssl_earliest_cert_expiry`. +Grafana is not a target, so nothing watches the one certificate this runbook +issues — expiry is still something you find out about from a browser warning. +Tracked in [#91](https://github.com/Gerrrt/HomeLab/issues/91). ## If something goes wrong