diff --git a/README.md b/README.md index b0de7ce..d21bde4 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,7 @@ git clone https://github.com/Gerrrt/HomeLab.git && cd HomeLab make secrets-init # generate an age keypair, create the encrypted secrets file make secrets-edit # fill in real values make certs ARGS=--ca # create the lab CA -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" # Grafana's leaf +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" # Grafana's leaf make validate # everything CI runs make up # render config and start the stack ``` diff --git a/docs/architecture.md b/docs/architecture.md index 1e2e537..3229e8d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -116,7 +116,7 @@ graph LR PROM[("Prometheus
:9090 · 30d")] LOKI[("Loki
:3100 · 30d")] AM["Alertmanager
:9093"] - GRAF["Grafana
:3000"] + GRAF["Grafana
:3000 · https"] end OUT([Webhook
notification]) @@ -175,7 +175,7 @@ re-shard of everything. Reasoning in | Service | Port | Bound to | Notes | | --- | --- | --- | --- | -| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only published service that authenticates | +| Grafana | 3000 | `${BIND_ADDR}` | The only UI meant to be opened by a human, and the only service that terminates TLS or authenticates — `https://`, on a lab-CA certificate a browser will warn about until you trust `certificates/ca.pem` | | Prometheus | 9090 | `${BIND_ADDR}` | Remote-write receiver — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | | Loki | 3100 | `${BIND_ADDR}` | Push endpoint — `oracle`'s agent pushes here. Unauthenticated; see [`security.md`](security.md) | | Alertmanager | 9093 | `127.0.0.1` | Nothing off-host uses it; silences are reached through Grafana | diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 5f9792d..bfc6196 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -23,7 +23,7 @@ make secrets-edit # replace every change-me value # TLS. Grafana serves https from this leaf and Prometheus verifies it with the # CA — see generate-certificates.md. Both are required before the stack starts. make certs ARGS=--ca -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" make validate # confirm the configs are sound before starting anything make up diff --git a/docs/runbooks/generate-certificates.md b/docs/runbooks/generate-certificates.md index 0afe918..59c6c1a 100644 --- a/docs/runbooks/generate-certificates.md +++ b/docs/runbooks/generate-certificates.md @@ -15,9 +15,11 @@ that rewrite as compromised, including the CA. What follows replaces it. > [`deploy-stack.md`](deploy-stack.md), not a someday step. `make render` > refuses to run until they exist, because before it did, `make up` bind-mounted > the absent files and Docker created directories in their place -> ([#69](https://github.com/Gerrrt/HomeLab/issues/69)). Prometheus, Alertmanager -> and Loki still publish plain HTTP on the management VLAN; putting the rest -> behind TLS is tracked in [`roadmap.md`](../roadmap.md). +> ([#69](https://github.com/Gerrrt/HomeLab/issues/69)). Grafana is the only +> service that terminates TLS. Prometheus, Alertmanager and Loki publish plain +> HTTP on the management VLAN, and that is an accepted residual with firewall +> default-deny as the control — not work in progress; see +> [`security.md`](../security.md). ## Where things live @@ -59,7 +61,7 @@ trust store holding it, so that has to be deliberate (`--force`). ## 2. Issue a leaf ```bash -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" ``` Include `--ip` for anything reached by address. `docs/roadmap.md` still lists @@ -67,6 +69,14 @@ internal DNS as unresolved, so in practice most services here are reached by IP, and a certificate without a matching IP SAN will be rejected at the point you most want it to work. +`--dns grafana` is not decoration. Prometheus scrapes the `grafana` job over the +compose network, where the service answers to its short name, so +[`prometheus.yaml`](../../stacks/observability/prometheus/prometheus.yaml) sets +`server_name: grafana` and verifies that name against the leaf. Issue this +certificate with the FQDN alone and Grafana still serves fine in a browser while +the scrape fails with `x509: certificate is valid for grafana.matrix.elysium, +not grafana` — one target down, for a reason that reads like a trust problem. + Two limits the script enforces rather than lets you discover later: - **A certificate with no `subjectAltName` is rejected outright** by every @@ -110,15 +120,17 @@ lab with one certificate is better served by a calendar reminder than by a renewal daemon nobody maintains. ```bash -make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --force" +make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana --force" ``` Then restart whatever serves it. The CA does not change, so nothing needs re-trusting. -Adding blackbox-exporter for TLS-expiry checks is on -[`roadmap.md`](../roadmap.md); until that exists, expiry is something you find -out about from a browser warning. +blackbox-exporter exists and could carry this, but does not yet: it probes only +the wiki, over plain HTTP, and nothing reads `probe_ssl_earliest_cert_expiry`. +Grafana is not a target, so nothing watches the one certificate this runbook +issues — expiry is still something you find out about from a browser warning. +Tracked in [#91](https://github.com/Gerrrt/HomeLab/issues/91). ## If something goes wrong @@ -146,7 +158,7 @@ certificate with `-s`, and `-s` is true of a directory: ```console $ ls -ld certificates/grafana.matrix.elysium-key.pem drwxr-xr-x 2 you you 4096 ... certificates/grafana.matrix.elysium-key.pem -$ make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20" +$ make certs ARGS="--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana" error: certificates/grafana.matrix.elysium-key.pem already exists — pass --force to replace it ``` diff --git a/scripts/gen-certs.sh b/scripts/gen-certs.sh index 1f78ba0..555373f 100755 --- a/scripts/gen-certs.sh +++ b/scripts/gen-certs.sh @@ -8,10 +8,12 @@ # The control that matters is not the file mode, it is that these never become # tracked files. `make validate` asserts that; so does CI. # -# Nothing in this repository terminates TLS yet. This exists so that when -# something does — Grafana is the first candidate, see docs/roadmap.md — issuing -# a certificate is one command rather than a research project, and so the CA is -# created deliberately rather than improvised at the point of need. +# Grafana terminates TLS with a leaf issued here, and Prometheus verifies that +# certificate against this CA when it scrapes the grafana job — so both are a +# prerequisite of `make up` rather than groundwork for a someday service. What +# this script buys is that issuing one stays a single command rather than a +# research project, and that the CA is created deliberately rather than +# improvised at the point of need. # # Usage: # scripts/gen-certs.sh --ca create the CA (refuses if it exists) @@ -66,7 +68,11 @@ while (($#)); do --dns) DNS+=("${2:?--dns needs a name}"); shift 2 ;; --days) LEAF_DAYS="${2:?--days needs a number}"; shift 2 ;; --force) FORCE=1; shift ;; - -h|--help) sed -n '2,32p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + # Printed straight from the comment block above rather than kept as a + # second copy that drifts. Bounded by where the comments stop, not by a + # line number: the literal 2,32p this replaced had already slipped and was + # truncating the last paragraph mid-sentence. + -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "${BASH_SOURCE[0]}"; exit 0 ;; *) die "unknown argument: $1" ;; esac done @@ -127,7 +133,7 @@ that holds it. Pass --force only if that is what you mean." ok "CA created" printf ' key %s (never commit, never copy off this host)\n' "${CA_KEY#"${REPO_ROOT}"/}" printf ' cert %s (safe to distribute — this is what clients trust)\n' "${CA_CRT#"${REPO_ROOT}"/}" - printf '\nIssue a leaf with:\n scripts/gen-certs.sh --host grafana.matrix.elysium --ip 10.0.99.20\n\n' + printf '\nIssue a leaf with:\n scripts/gen-certs.sh --host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana\n\n' exit 0 fi diff --git a/scripts/render-config.sh b/scripts/render-config.sh index c0e7e4e..6d00ad4 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -93,7 +93,7 @@ Grafana serves https from the leaf and Prometheus verifies it with the CA, so the stack cannot start without them. Generate them with: make certs ARGS=--ca - make certs ARGS=\"--host grafana.matrix.elysium --ip 10.0.99.20\" + make certs ARGS=\"--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana\" Full procedure in docs/runbooks/generate-certificates.md." fi diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 356f7ba..16ea909 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -11,7 +11,7 @@ make up # from the repository root | `prometheus` | `prom/prometheus` | 9090 | Metrics store, remote-write receiver, rule evaluation | | `alertmanager` | `prom/alertmanager` | 9093 (localhost) | Alert routing, grouping, inhibition | | `loki` | `grafana/loki` | 3100 | Log store | -| `grafana` | `grafana/grafana-oss` | 3000 | Dashboards — the only published UI, and the only one that authenticates | +| `grafana` | `grafana/grafana-oss` | 3000 (https) | Dashboards — the only published UI, and the only service that terminates TLS or authenticates | | `snmp-exporter` | `prom/snmp-exporter` | *internal* | SNMP polling proxy | | `alloy` | `grafana/alloy` | 12345 (localhost) | Metric and log collection |