Probe the services this was filed for, and probe TLS expiry

[Gerrrt]

Nothing checks that internal services answer, or that their certificates are
still valid.

docs/runbooks/generate-certificates.md:106 states the consequence plainly:

Adding blackbox-exporter for TLS-expiry checks is on roadmap.md; until that
exists, expiry is something you find out about from a browser warning.

Current expiries, for reference — not urgent, which is the argument for building
this before it is:

Certificate	Expires	Days
grafana.matrix.elysium	2028-11-21	823
Matrix Elysium Internal CA	2036-08-16	~3650

gen-certs.sh caps leaves at 825 days, so a leaf issued today will always be
inside blackbox's useful range.

What it should probe: Grafana (HTTPS, expiry, chain against the lab CA),
Prometheus, Alertmanager, Loki, the switch management UI, the iLO, and the
pfSense web UI. That also gives an availability signal for the four devices that
currently only have SNMP.

Tracked in docs/roadmap.md; filed as an issue so it has a place to be discussed and closed.

[Gerrrt]

Two alternatives worth knowing about before building this, from the selfh.st directory — both cover the same ground as blackbox-exporter from a different angle.

Gatus Apache-2.0 (~11.9k stars) — declarative YAML health checks with conditions, alerting and a status page. HTTP, TCP, ICMP, DNS, and certificate expiry are all first-class.

The reason it is worth a look here rather than being dismissed as duplication: its configuration is a file, which is the property this repository keeps choosing on purpose — provisioned dashboards, alert rules in YAML, generator.yaml as the source of truth for SNMP. A Gatus config diffs in git log and validates in CI the same way everything else here does. It also produces a status page, which blackbox-exporter does not, and #124 (no entry point for a successor operator) suggests a page saying what should be up has some value beyond alerting.

Uptime Kuma (~90.6k stars) is the popular answer and is the wrong shape for this repo: everything lives in its database, so the checks would exist only there. That is precisely the failure ADR-0004 rejected for Grafana dashboards — "No dashboard exists only in a database". Noting it mainly so it can be declined on the record rather than proposed again later.

That said, blackbox-exporter is probably still correct here, for a specific reason: #126 needs the AdGuard probe to feed a Prometheus rule that routes through the existing Alertmanager tree, with inhibition and severity routing already built. Gatus would run its own alerting path in parallel, which means two notification paths to keep working — and #67 is a recent reminder of what an unwatched notification path does.

So the useful framing may be: blackbox-exporter for anything that should alert, Gatus only if a human-readable status page is wanted as well. Adding both to alert is the option to avoid.

---

Generated by Claude Code

[Gerrrt]

Status check while fixing #74 — the roadmap entry for this was stale, and not in the direction you'd expect.

blackbox-exporter is deployed. compose.yaml:258-279 defines it, prometheus.yaml:55-57,105-123 scrapes it and runs the blackbox job, targets/blackbox.yaml has targets. So "add blackbox-exporter" is done.

But none of the seven services above are probed. Grafana, Prometheus, Alertmanager, Loki, the switch UI, the iLO, the pfSense UI — none. What is probed is the wiki, twice (by name and by address), which this issue never mentions; it was added later after the wiki went unreachable unnoticed.

The expiry half has nothing behind it, verified rather than assumed:

• No rule reads probe_ssl_earliest_cert_expiry — grepped ssl, cert, expiry, expir across all six files in prometheus/rules/, zero hits.
• Both targets use http_2xx_plain against http:// URLs, so no TLS handshake happens and the metric is never produced.
• The TLS-capable http_2xx module (blackbox/blackbox.yaml:27-42) is defined and referenced by nothing.
• Grafana — the only service in the estate terminating TLS — is not a target, so there is not even a series to alert on.

The table above still holds: grafana.matrix.elysium expires 2028-11-21, the CA 2036-08-16. Nothing is close, which is still the argument for building this before it is.

Two docs corrected to match, both of which claimed this component did not exist:

• docs/roadmap.md — docs(roadmap): say what #91 still needs, now that the exporter exists #198
• docs/runbooks/generate-certificates.md Renewal section, which said expiry checks were pending "until that exists" — folded into docs(tls): say Grafana terminates TLS, and issue a leaf that verifies #196

Nothing moved to the roadmap's Done list: that file's intro says whether a thing is started, blocked or done lives on its issue, and "two places holding the same checkbox is how a checkbox stops being true".