Upgrade Prometheus to 3.13.2 LTS, Alertmanager to 0.34.0, Node Exporter to 1.12.1

[jpmckinney]

Annual Prometheus upgrade, following the documented process. Prometheus 3.13 is the current LTS (supported until 2027-07-31); the 3.5 LTS reached end of support on 2026-07-31.

One commit per change:

1. Upgrade Prometheus 3.5.0 → 3.13.2, Alertmanager 0.28.1 → 0.34.0, Node Exporter 1.9.1 → 1.12.1 (pillar/prometheus_client.sls, pillar/prometheus_server.sls).
2. Thread email notifications from the same alert group (Alertmanager 0.30+), so repeated notifications stack in one mailbox thread instead of scattering.
3. Cap TSDB disk usage at 70% of the filesystem (Prometheus 3.11+), as a backstop so Prometheus' own data cannot fill the disk. Also moves the retention time from the --storage.tsdb.retention.time flag (deprecated in 3.13) to the configuration file, so retention changes take effect on reload instead of restart.
4. Alert when a filesystem is remounted read-only — Node Exporter 1.10 made node_filesystem_readonly reliable by taking superblock options into account.

Stale time series of deleted servers are left to expire on the retention schedule, rather than enabling the TSDB admin API for Prometheus 3.12's series deletion web interface (see comments).

The rendered conf-prometheus.yml, conf-prometheus-rules.yml (22 rules), and conf-alertmanager.yml were validated with promtool check config (3.13.2) and amtool check-config (0.34.0). The breaking-changes and new-features reviews follow in separate comments.

Deploy notes: upgrading the server and Alertmanager requires deploying the prometheus target; upgrading Node Exporter requires deploying all targets.

🤖 Generated with Claude Code

https://claude.ai/code/session_013pw9DLYCn7sGHF197pKY7C

[jpmckinney]

Breaking changes review

Every [CHANGE] entry in the changelogs between our versions and the targets was checked against what this repository deploys (pillar files, conf-prometheus.yml, conf-alertmanager.yml, the alert rules, and the three systemd unit templates). None affects our configuration:

Prometheus 3.5.0 → 3.13.2 (changelog)

• 3.13.0: Credentials are no longer forwarded when following a redirect to a different host (CVE-2025-4673, CVE-2023-45289). Our scrapes are direct static targets over HTTPS with basic auth — no redirects involved.
• 3.13.0: npm_licenses.tar.bz2 removed from release tarballs. The Salt state only extracts and runs the binary; tarball and sha256sums.txt URLs for 3.13.2 were verified to exist.
• 3.13.0: Rule group pagination tokens use SHA-256 — API detail, nothing consumes them.
• 3.11.0: Hetzner SD label deprecations — we use static configs on Linode.
• 3.9.0: Native histograms stable, native-histogram feature flag a no-op — not used.
• 3.8.0: Remote-write receiving updated to 2.0-rc.4 spec; 3.7.0: remote-write metrics deprecated — remote write/read not used.
• 3.13: --storage.tsdb.retention.time flag deprecated (still functional) — this PR moves it to the configuration file.
• All flags in prometheus-server.service (--web.listen-address, --config.file, --storage.tsdb.path) remain stable.

Alertmanager 0.28.1 → 0.34.0 (changelog)

• 0.33.0: --enable-feature=auto-gomaxprocs flag removed — not passed in prometheus-alertmanager.service.
• 0.33.0: alertmanager_marked_alerts metric removed — not referenced in our alert rules. ⚠️ If any Grafana dashboard (stored on the server, not in this repository) charts this metric, it needs a manual tweak.
• 0.34.0: reason label on alertmanager_notifications_failed_total now distinguishes authError and rateLimited from clientError — not matched anywhere in this repository; same Grafana caveat applies.
• 0.32.0: UI assets no longer checked into the repository — build-from-source concern only; release tarballs are unaffected.

Node Exporter 1.9.1 → 1.12.1 (changelog)

• 1.10.0: mdadm metrics read from sysfs; erofs added to default filesystem excludes — our rules only use node_cpu/node_disk/node_filesystem/node_memory/node_textfile metrics; no node_md_* usage, no erofs.
• Flags in prometheus-node-exporter.service (--web.config.file, --collector.textfile.directory) remain stable.

Data on disk (the documented concern)

The server upgrade stays within the 3.x major version, and no TSDB, silence, or notification-log format changes appear in any of the changelogs, so the data in /home/prometheus-server/data/ and the Alertmanager state carry over. Standard caveat: after upgrading, downgrading back across several minor versions is not guaranteed. Node Exporter is stateless.

---

Generated by Claude Code

[jpmckinney]

New features review

Features and enhancements between our versions and the targets that are relevant to this deployment. The first four are adopted in this PR; the rest need no configuration.

Adopted in this PR

• Email threading (Alertmanager 0.30.0). threading: enabled on email_configs makes notifications from the same alert group share an email thread, grouped daily (thread_by_date: daily — Alertmanager requires it to be explicit when the threading block is present; validated with amtool check-config). With our repeat_interval: 1d, repeated firings now stack in one thread.
• Percentage-based retention (Prometheus 3.11.0). storage.tsdb.retention.percentage: 70 caps the TSDB at 70% of the filesystem, adapting automatically if the volume is resized. Time-based retention (120d) remains the primary policy; the cap is a backstop so Prometheus cannot trip our own RootFileSystemLow alert on the monitor host. Marked experimental upstream. Relatedly, 3.13.2's headline fix prevents a SIGBUS crash when the data disk fills — the exact failure mode this guards against.
• Reliable read-only filesystem detection (Node Exporter 1.10.0). node_filesystem_readonly now accounts for superblock options, so a new FileSystemReadOnly alert catches emergency read-only remounts — the classic symptom of underlying disk trouble.
• Series deletion web interface (Prometheus 3.12.0). The TSDB status page can delete time series and clean tombstones. It calls the TSDB admin API, so --web.enable-admin-api is now set (the endpoints sit behind the same Apache basic authentication as the rest of the web interface — note this also exposes snapshot and WAL admin endpoints to authenticated users). docs/deploy/delete_server.rst gains a step to purge a decommissioned server's series.

Available with no configuration

• Silence improvements (Alertmanager 0.32.0–0.34.0): silences support annotations (a "why" note), multiple matcher sets per silence, and server-side state filters in the API.
• Stricter email configuration validation (Alertmanager 0.31.0): SMTP misconfigurations are caught at startup.
• Config auto-reload promoted to stable (Prometheus 3.12.0): not needed — Salt already reloads the service when configuration files change, and retention is now runtime-reloadable via the config file.
• "Unknown" state for not-yet-evaluated alerting rules (Prometheus 3.8.0) and various UI improvements.

Not relevant to this deployment: native histograms, remote-write 2.0, OTLP ingestion, cloud service discovery (AWS/Azure/Hetzner/Outscale), and chat integrations (Slack/Mattermost/Telegram/Discord).

---

Generated by Claude Code

[jpmckinney]

The commit that enabled the TSDB admin API and documented series deletion in delete_server.rst has been removed from this PR. Deleting time series (via Prometheus 3.12's web interface or otherwise) requires --web.enable-admin-api, which also exposes snapshot and WAL-truncation endpoints.

Instead, we rely on the retention schedule: a deleted server's series stop receiving samples immediately, so they only occupy historical blocks and expire within the 120-day retention window — now also bounded by the 70% disk-usage cap added in this PR.

---

Generated by Claude Code

[jpmckinney]

Deployed all except registry, as scrapers are running.