From 2d13027aedbe42e7bf3cbcc97a3f2de186187452 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 31 Aug 2026 01:31:23 +0000 Subject: [PATCH] fix(observability): reap orphans, bound tasks, cap the TSDB (#71) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #71 records that no service sets any resource bound and that Prometheus retention is time-only. Its memory half is #114 and stays there — the nine days of history since have widened the case rather than settled it, so the roadmap entry is rewritten to say that waiting for more is not what unblocks it. This is the half that was sizeable. Verifying it turned up a live fault. Grafana was holding 2437 defunct ssl_client processes and gaining two more every 30 seconds: busybox wget forks a TLS helper for the https healthcheck, the helper is reparented to PID 1 on exit, and PID 1 there is `grafana server`, which never reaps. At 2880 charges a day against the inherited pids.max of 9056 it would have run out of tasks about three days after each start, presenting as `unhealthy` — which Docker does not restart. It was only ever saved by being redeployed sooner than that. Nothing in the stack could see it: container_tasks_state reads zero for every state. So `init: true` comes first and `pids_limit` second; the reverse order would have cut Grafana off sooner rather than protecting it. pids_limit is 512 (1024 for alloy, 4096 for the unmeasured renderer) against an observed 9-17 tasks and a 21-thread peak. The number is loose deliberately: what makes a task ceiling safe to set and a memory limit not is that the gap between "certainly enough" and "certainly protective" spans two orders of magnitude here. ulimits stays absent, and the file says why — RLIMIT_NPROC is enforced per real UID across the whole kernel, and alertmanager and snmp-exporter run as the operator's own uid. The TSDB ceiling is 12 GiB, sized from the per-block write rate rather than the store's current size. 1021 MiB on disk is not a steady state: prometheus_tsdb_time_retentions_total is 0, nothing has ever been evicted, and the rate went 16.1 -> 72.9 MiB/day at 08-17 when #62 fixed cAdvisor and the series count went 3,514 -> 24,005. Steady state is 2.28 GiB, or 2.73 GiB once Saruman gets an agent, so 4 GiB would have been 1.5x against an estate that is already planned — a second retention policy wearing a ceiling's clothes, in a repository that promises 30 days in four places. A ceiling that engages silently shortens retention to whatever fits, so PrometheusSizeRetentionActive says when it does, keyed on the size counter rather than the time counter, which increments constantly by design. It ships with promtool cases for both firing and quiet, per #63. Deployed to the lab in two cycles and confirmed: init=true on all seven, pids.max 512/1024 read back from the kernel, retention_limit_bytes 12884901888, grafana flat at 20-29 tasks with zero defunct children across five healthcheck intervals, 39 rules loaded and healthy, no OOM kills and no restarts. Co-Authored-By: Claude Opus 5 --- README.md | 6 +- docs/observability.md | 15 +++- docs/roadmap.md | 21 +++-- docs/runbooks/deploy-stack.md | 39 ++++++++- docs/security.md | 9 +- stacks/observability/.env.example | 10 +++ stacks/observability/README.md | 2 +- stacks/observability/compose.yaml | 86 +++++++++++++++++++ .../prometheus/rules/containers.rules.yaml | 26 ++++++ .../prometheus/tests/containers.test.yaml | 42 +++++++++ 10 files changed, 241 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index de6a425..b0de7ce 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 5 provisioned dashboards, 84 panels, and - 51 alert rules — 38 metric-based in Prometheus, 13 log-based in Loki — sharing + 52 alert rules — 39 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -138,7 +138,7 @@ rack; a dashed border means egress only. Full topology and data flow in | --- | --- | --- | | Firewall / routing | [pfSense on FreeBSD 15](docs/network.md) | VLANs, DHCP, default-deny | | Virtualisation | Proxmox VE | Lab hypervisor | -| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention, remote-write receiver | +| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention capped at 12 GiB, remote-write receiver | | Logs | [Loki](stacks/observability/loki) | Single-binary, filesystem storage | | Collection | [Grafana Alloy](stacks/observability/alloy) | node + cAdvisor metrics, Docker/journal/syslog/auth logs | | Network polling | [snmp_exporter](stacks/observability/snmp-exporter) | pfSense, switch, UPS, iLO | @@ -153,7 +153,7 @@ rack; a dashed border means egress only. Full topology and data flow in . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 38 alert rules +│ ├── prometheus/ # config, file_sd targets, 39 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # one agent config, used on every host diff --git a/docs/observability.md b/docs/observability.md index aedc223..ff61501 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -22,6 +22,15 @@ Retention is 30 days for both metrics (`PROMETHEUS_RETENTION` in `.env`) and logs (`retention_period` in `loki/loki-config.yaml`). Change both together or dashboards will show metrics with no matching logs at the far end of the range. +Metrics carry a second bound: `PROMETHEUS_RETENTION_SIZE` caps the store at +12 GiB, and whichever limit is reached first wins. It is sized from the write +rate rather than the store's current size — 72.9 MiB/day gives 2.28 GiB over +30 days, 2.73 GiB once `Saruman` gets an agent — so it is roughly 4.4x the +planned estate and should never bind. In normal operation the 30 days above is +the limit that applies; if the size cap ever binds, that 30 days stops being +true and `PrometheusSizeRetentionActive` is what says so. Loki has no +equivalent size bound. + ## Log level normalisation Logs arrive spelling severity about twenty different ways — `ERROR`, `err`, @@ -70,7 +79,7 @@ expression in every panel is syntactically valid. ## Alerting -51 rules in total: 38 metric-based in `prometheus/rules/`, and 13 log-based in +52 rules in total: 39 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -99,7 +108,7 @@ boot check. ### Metric-based (Prometheus) -38 rules across six files in `prometheus/rules/`: +39 rules across six files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -117,7 +126,7 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is two rules of 35 so far: +against the broken rule too. Coverage is six rules of 39 so far: `ContainerHighMemory` and `Watchdog`. The other 33 are still validated for syntax only, which is exactly the standing #63 had. diff --git a/docs/roadmap.md b/docs/roadmap.md index af59aaf..7546766 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -54,12 +54,21 @@ issues intact. Nothing was summarised away. something you find out about from a browser warning. - **[#114](https://github.com/Gerrrt/HomeLab/issues/114) Set memory limits on the six services.** Nothing in `compose.yaml` bounds a leak, so one container - can take the host down — and the host has 8 GB soldered. Blocked on data - rather than on agreement: cAdvisor has only reported correctly since - [#62](https://github.com/Gerrrt/HomeLab/pull/62), so there are hours of - history to size from, not the 30 days retention implies, and `grafana` alone - swings 3.7x inside that window. A limit picked from it would be a guess at an - OOM kill. + can take the host down — and the host has 8 GB soldered. It was blocked on + data: cAdvisor has only reported correctly since + [#62](https://github.com/Gerrrt/HomeLab/pull/62), and `grafana` swung 3.7x + inside the six hours available, so a limit picked from it would have been a + guess at an OOM kill. Nine more days did not settle it — they widened it. + `loki` now swings 8.6x (121 MiB median, 1039 MiB peak), and `loki`, `grafana` + and `alloy` all peaked in the *same hour* on 2026-08-29, which is an episodic + event rather than a distribution that converges with more sampling. The + method the issue proposed no longer fits the machine either: 3x every peak is + 7536 MiB against 7816 MiB of RAM. So the gate has changed rather than moved — + waiting for more history is not what unblocks this, explaining that one hour + is. [#71](https://github.com/Gerrrt/HomeLab/issues/71) took the half that was + sizeable: `pids_limit`, because tens of threads against a 10,000-thread abort + is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which + is at a measurable steady state at day 28 of 30. - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard screenshots.** `make screenshots` does four of the five; the Logs dashboard is deliberately excluded. → [`images/README.md`](images/README.md) diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index bba9779..5f9792d 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -63,11 +63,48 @@ ss -ltn | grep -E ':(9090|3100|3000)' # BIND_ADDR — 0.0.0.0 by default ss -lun | grep ':1514' # BIND_ADDR ``` +Confirm the per-container limits applied too. A limit that silently failed to +take is the thing they exist to defend against, and nothing in the stack would +report it (#71): + +```bash +for c in prometheus alertmanager loki grafana snmp-exporter blackbox-exporter alloy; do + docker inspect "$c" --format '{{.Name}} init={{.HostConfig.Init}} pids={{.HostConfig.PidsLimit}}' +done +``` + +Every service should read `init=true`, and `pids=512` except `alloy`, which is +`1024`. That reads back what Docker was *asked* for, so confirm the kernel +agrees — a stale cgroup directory will happily show you the old ceiling: + +```bash +p=$(docker inspect -f '{{.State.Pid}}' prometheus) +cat "/sys/fs/cgroup$(cut -d: -f3 /proc/$p/cgroup | head -1)/pids.max" # 512 +``` + +`init=false` on grafana means the zombie leak is back: + +```bash +docker exec grafana sh -c 'ls -d /proc/[0-9]* | wc -l' +``` + +which should be single digits and stay there, not climb by two every 30 seconds. + +Confirm the size ceiling is armed, too. `PrometheusSizeRetentionActive` only +fires once the ceiling *bites*, so if the flag were ever dropped this would +silently read `0` and the rule would stay quiet forever — the #63 shape: + +```bash +curl -s 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_retention_limit_bytes' +``` + +`12884901888` is 12 GiB. `0` means there is no size bound in force. + Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 38 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 39 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Five dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/security.md b/docs/security.md index 2b52868..5b4b803 100644 --- a/docs/security.md +++ b/docs/security.md @@ -14,7 +14,7 @@ What this network is actually built to survive: | A smart TV's firmware phoning somewhere unexpected | VLAN 40 is terminal, egress only | | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | -| Losing visibility of a failure | 38 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 39 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | @@ -243,6 +243,13 @@ complete state table and interface topology. They are credentials. [ADR-0012](adr/0012-publish-only-ports-with-an-off-host-consumer.md). - The Alloy debug UI binds to `127.0.0.1` only. - The Docker socket is mounted read-only into Alloy. +- Every service runs under a real init (`init: true`) and a chosen task ceiling + (`pids_limit`, 512; 1024 for Alloy) rather than the inherited systemd default + of 9056. This was not theoretical: Grafana's https healthcheck was leaking two + unreaped `ssl_client` children every 30 seconds and would have exhausted the + inherited ceiling about three days after each start (#71). +- Prometheus carries a byte ceiling as well as a time one, so a change that + quietly multiplies the series count cannot consume the disk unnoticed. - All images are pinned to explicit versions, so an upstream compromise cannot arrive silently via `:latest`. Dependabot proposes the bumps; CI validates them. diff --git a/stacks/observability/.env.example b/stacks/observability/.env.example index a1ee1e5..e5e4109 100644 --- a/stacks/observability/.env.example +++ b/stacks/observability/.env.example @@ -40,3 +40,13 @@ SYSLOG_PORT=1514 # Metric retention. Loki's log retention is set separately in # loki/loki-config.yaml (retention_period) — keep the two in step. PROMETHEUS_RETENTION=30d + +# A byte ceiling on the same store, so whichever limit is reached first wins. +# Sized from the per-block write rate, not from the store's current size: at +# 72.9 MiB/day, 30 days is 2.28 GiB, or 2.73 GiB once Saruman gets an agent +# (#88). 12 GiB is 4.4x that and 12% of the volume. It is meant never to bind — +# if it does, retention is quietly shorter than the line above claims, and +# PrometheusSizeRetentionActive is what says so. compose.yaml has the full +# derivation and the reason a smaller number would have been wrong. Loki has no +# size bound at all. +PROMETHEUS_RETENTION_SIZE=12GB diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 05e401d..356f7ba 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox.yaml probe targets — hot-reloaded, no restart needed - rules/*.rules.yaml 38 alert rules across host/network/ups/containers/blackbox + rules/*.rules.yaml 39 alert rules across host/network/ups/containers/blackbox tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability from outside the service alertmanager/ diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 125217e..0710df9 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -20,6 +20,44 @@ x-service-defaults: &service-defaults restart: unless-stopped logging: *default-logging networks: [observability] + # A real init as PID 1, so orphaned children are reaped. Not hygiene: on + # 2026-08-31 grafana was holding 2437 defunct `ssl_client` processes and + # gaining two more every 30 seconds. Busybox wget forks a TLS helper for the + # https healthcheck further down, the helper is reparented to PID 1 when wget + # exits, and PID 1 in that container is `grafana server` — which never calls + # wait(). The default pids.max is 9056, so it ran out of tasks about three + # days after each start, and was only ever saved by being redeployed sooner + # than that. Every other healthcheck here is plain HTTP and forks nothing, + # which is why grafana was the only one that showed it (#71). + init: true + # Bounds a task leak of the kind the paragraph above describes. Steady state + # is 9-17 tasks per service and the highest go_threads seen over nine days is + # 21, so 512 is ~25x normal and still well under the Go runtime's own + # 10,000-thread abort. The number is loose on purpose: what makes a PID limit + # safe to set — and a memory limit not, which is why there is no mem_limit in + # this file and #114 owns that question — is that the gap between "certainly + # enough" and "certainly protective" spans two orders of magnitude here. A + # container that hits this dies on EAGAIN, restart: unless-stopped recovers + # it, and ContainerRestartLoop fires. The quieter case is the one that + # actually happened: the app was fine and the *healthcheck* could not fork, + # which shows as `unhealthy` and is not something Docker restarts — the same + # gap this file notes on loki. There is no rule watching PID counts directly + # because cAdvisor is not exporting container_processes; Alloy's exporter + # disables the "process" metric kind, and turning it on means restating the + # whole disabled list, since enabled_metrics overrides it rather than adding + # to it. Getting that wrong drops every other container metric, which is the + # #62 shape exactly. Worth its own issue, not a line here. + # + # `ulimits` is absent, and deliberately. nofile would throttle the mmap'd + # chunk files prometheus and loki depend on, to defend against an exhaustion + # never observed here. nproc is worse than useless: RLIMIT_NPROC is enforced + # per real UID across the whole kernel, not per container, and alertmanager + # and snmp-exporter both run as ${RENDER_UID} — the operator's own uid, set + # from `id -u` by scripts/render-config.sh. An nproc there would count the + # operator's host processes against the container, and could refuse the + # operator a login shell. pids_limit is a cgroup control and is scoped + # correctly; ulimits: nproc is a uid control in a container costume. + pids_limit: 512 services: # --------------------------------------------------------------------------- @@ -35,6 +73,43 @@ services: - --config.file=/etc/prometheus/prometheus.yaml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-30d} + # A ceiling on the TSDB, not a tidy-up. It bounds *sustained* growth — + # the series count creeping up as hosts and exporters are added — so that + # the disk cannot be consumed a few MiB at a time by a change nobody + # thought of as a storage decision. + # + # Do not size it from the store's current size. On 2026-08-31 that was + # 1021 MiB, and prometheus_tsdb_time_retentions_total was 0 — nothing had + # ever been evicted, so the store was still filling and that number was + # not a steady state. Per-block rates over the retained window: 16.1 + # MiB/day up to 08-17, then 72.9 MiB/day after, because #62 fixed cAdvisor + # and the series count went 3,514 -> 24,005. Steady state at the current + # rate is 30 x 72.9 + ~150 MiB of WAL and head chunks = 2.28 GiB, and + # ~2.73 GiB once Saruman gets an agent (#88). So 12GB — powers-of-2, i.e. + # 12 GiB — is 5.3x today and 4.4x with #88, and 12% of the 98 GiB volume. + # 4 GiB looked generous against 1021 MiB and would have been 1.5x against + # the estate that is already planned. + # + # That headroom is the point: this repository promises 30 days in four + # places, and a ceiling reached in normal operation is a second retention + # policy wearing a ceiling's clothes. If it ever does evict, + # PrometheusSizeRetentionActive in containers.rules.yaml says so. + # + # What it does NOT do is win a race against a flood through the + # remote-write receiver enabled below, which anything that can route here + # may push to (ADR-0012, #182). The budget counts the WAL and the m-mapped + # head chunks but only ever evicts persistent blocks, and a flood lands in + # the WAL: at a few MB/s the head alone passes 12 GiB long before any + # block does, at which point every block is deleted and the WAL keeps + # growing regardless. Bounding that is authentication's job, not this + # flag's. + # + # Both retention flags are marked [DEPRECATED] by this version in favour + # of a storage.tsdb.retention block in prometheus.yaml. They still work, + # and they stay here because prometheus.yaml is mounted rather than + # rendered — moving retention into it would drop these two tunables out of + # .env.example, which is where this repository deliberately keeps them. + - --storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE:-12GB} - --web.enable-remote-write-receiver - --web.enable-lifecycle volumes: @@ -321,6 +396,10 @@ services: # from the environment instead; the header of that file has the whole story. image: grafana/alloy:v1.19.2@sha256:b8ec653c44235fbe910879145dac3597d66b0aaecf60bcbbe82580767771a839 container_name: alloy + # 1024 rather than the shared 512: this is the one service that walks + # cgroups, tails the journal and talks to the Docker socket, so it has more + # reason than the others to fan out. Observed peak is 20 threads either way. + pids_limit: 1024 privileged: true # Without this the cAdvisor exporter reports exactly one series, id="/", # and every per-container metric is missing — which is what the Docker @@ -414,6 +493,13 @@ services: # through a large dashboard, which presents as a truncated PNG rather than # an error. shm_size: "1gb" + # Overrides the shared 512, which is a number measured from Go services and + # means nothing here — a headless Chromium is a process tree, not one + # binary. This value is not measured either: the `capture` profile is off on + # every normal deploy, so there was nothing running to count. It is set + # rather than inherited so that the shared default cannot quietly break a + # `make screenshots` run, and it is high enough that it should not bind. + pids_limit: 4096 # --------------------------------------------------------------------------- # tar and gzip for scripts/backup-volumes.sh. NOT part of the running stack: diff --git a/stacks/observability/prometheus/rules/containers.rules.yaml b/stacks/observability/prometheus/rules/containers.rules.yaml index 234497b..a7fa2f0 100644 --- a/stacks/observability/prometheus/rules/containers.rules.yaml +++ b/stacks/observability/prometheus/rules/containers.rules.yaml @@ -89,6 +89,32 @@ groups: annotations: summary: "{{ $value }} rule evaluation failures in the last 15 minutes" + - alert: PrometheusSizeRetentionActive + # --storage.tsdb.retention.size is a ceiling that, when it engages, + # silently shortens retention from the 30 days everything else in this + # repository claims to whatever fits in 4 GiB. Nothing else would say + # so: the store keeps working, the dashboards keep drawing, and you find + # out when a query over last month comes back short. Prometheus counts + # size-based eviction separately from time-based eviction, which is the + # only thing that distinguishes "the ceiling bound" from "old data aged + # out normally" — do not substitute prometheus_tsdb_time_retentions_total + # here, it increments constantly and by design. + # + # Firing means one of two things and both want a human: the series + # budget grew past what 4 GiB holds for 30 days, or something is pushing + # volume into the open remote-write receiver. Either way the documented + # retention is no longer true. + expr: increase(prometheus_tsdb_size_retentions_total[1h]) > 0 + labels: + component: containers + severity: warning + category: capacity + annotations: + summary: "Prometheus is evicting blocks to stay under its size ceiling" + description: >- + Retention is now bounded by PROMETHEUS_RETENTION_SIZE rather than + PROMETHEUS_RETENTION, so history is shorter than 30 days. + - alert: AlertmanagerNotificationsFailing expr: increase(alertmanager_notifications_failed_total[15m]) > 0 for: 5m diff --git a/stacks/observability/prometheus/tests/containers.test.yaml b/stacks/observability/prometheus/tests/containers.test.yaml index b08b860..5f968b5 100644 --- a/stacks/observability/prometheus/tests/containers.test.yaml +++ b/stacks/observability/prometheus/tests/containers.test.yaml @@ -148,3 +148,45 @@ tests: category: capacity exp_annotations: summary: "Container alloy is using 50% of host memory" + + # --- PrometheusSizeRetentionActive: fires when the ceiling engages -------- + # The counter is flat until the size ceiling starts evicting blocks, then it + # increments once per eviction. One step inside the hour is the whole signal. + - interval: 1m + input_series: + - series: 'prometheus_tsdb_size_retentions_total{instance="localhost:9090",job="prometheus"}' + values: "0+0x40 1+0x30" + alert_rule_test: + - eval_time: 60m + alertname: PrometheusSizeRetentionActive + exp_alerts: + - exp_labels: + alertname: PrometheusSizeRetentionActive + instance: localhost:9090 + job: prometheus + component: containers + severity: warning + category: capacity + exp_annotations: + summary: "Prometheus is evicting blocks to stay under its size ceiling" + description: >- + Retention is now bounded by PROMETHEUS_RETENTION_SIZE rather + than PROMETHEUS_RETENTION, so history is shorter than 30 days. + + # --- PrometheusSizeRetentionActive: quiet while the ceiling is not bound -- + # This is the production state and the one the rule must not alert on: the + # counter exists and reads 0 forever. Without this case the rule would pass + # its test while being wired to a metric that is always absent. + - interval: 1m + input_series: + - series: 'prometheus_tsdb_size_retentions_total{instance="localhost:9090",job="prometheus"}' + values: "0x70" + # Time-based eviction runs constantly and by design. If the expression is + # ever "simplified" onto this counter instead, the rule fires every hour + # forever, and this case is what catches that. + - series: 'prometheus_tsdb_time_retentions_total{instance="localhost:9090",job="prometheus"}' + values: "0+1x70" + alert_rule_test: + - eval_time: 60m + alertname: PrometheusSizeRetentionActive + exp_alerts: []