Skip to content

fix(ci): guard the counted claims the checker was still missing - #195

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/stale-alert-panel-counts-853424
Aug 31, 2026
Merged

fix(ci): guard the counted claims the checker was still missing#195
Gerrrt merged 1 commit into
mainfrom
gerrrt/stale-alert-panel-counts-853424

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Follow-up to the close-out of #72. Two gaps found while verifying that issue, both recorded in its closing comment.

Neither was a stale count. scripts/check_docs.py already derives the counts from the configs, and every number in the repository is currently correct. These are holes in the guard itself — claims it could not see, which would therefore drift silently. That is the failure #72 documents, surviving inside its own fix.

Three unguarded claims

Where Claim Why it was invisible
README.md:158 13 LogQL rules The Loki pattern is log-based. One line from a checked "13 log-based", and matched nothing.
docs/observability.md:111 39 rules across six files Two counts on one line. Only the first was checked, so splitting a rule file could not fail here.
docs/images/README.md:50,84 five dashboards Present-tense, but not in PROSE at all.

The last two are spelled out, which is what made them invisible — the checker only matched digits. Counts are now matched either way. The word list stops at twenty to keep the alternation short (not a claim about the ceiling) and is ordered longest-first so seven cannot match inside seventeen and leave the rest of the pattern to fail.

Bringing docs/images/README.md into PROSE also subjects it to the other five assertions. It passes them unchanged, and its dashboard claims are accurate today: 5 dashboard JSONs, 4 PNGs, homelab-logs excluded on purpose in capture-screenshots.sh:56-60.

Mutation tests

Each new assertion was tested in both directions, following the convention b316399 set.

Changing the prose fails:

README.md:158 claims 12 LogQL rules; the repository has 13
docs/observability.md:111 claims seven Prometheus rule files; the repository has 6
docs/images/README.md:84 claims six dashboards; the repository has 5

Changing the configs the prose describes fails too — which is the direction that actually matters, because it is how the Suricata, firewall-log and UPS self-test rules went stale in the first place. A seventh rule file carrying no new alerts trips the file count on its own:

docs/observability.md:111 claims six Prometheus rule files; the repository has 7

and a sixth dashboard trips both previously-unguarded lines:

docs/images/README.md:50 claims five dashboards; the repository has 6
docs/images/README.md:84 claims five dashboards; the repository has 6

Adding one real alert rule produces ten failures across six files, including docs/runbooks/deploy-stack.md:107 — the deploy-runbook step that made #72 worth filing.

make check-docs

check_docs.py was the only check reachable from validate.sh and CI but not the Makefile. That is the same asymmetry #68 recorded for snmp-targets.sh, which b316399 fixed on the CI side; this closes it on the Makefile side. Placed before check-dashboards to match the order CI runs them in.

Verification

./scripts/validate.sh passes end to end, including gitleaks over the full history and the lint that asserts ci.yml, the Makefile and validate.sh all delegate to the same scripts.

docs OK — 39 Prometheus + 13 Loki rules, 5 dashboards, 84 panels, 6 assertions
all checks passed

No document or config changed — only the checker and the Makefile.

Refs #72, #68


Generated by Claude Code

scripts/check_docs.py closed #72 by deriving the counts from the configs,
but it only ever looked for the phrasings someone happened to think of.
Three present-tense claims sat outside it, all correct and all able to go
stale silently — the failure #72 is about, surviving inside its own fix.

  README.md:158              "13 LogQL rules" — the Loki pattern is
                             `log-based`, so this matched nothing, one
                             line away from a checked "13 log-based"
  docs/observability.md:111  "39 rules across six files" states two
                             counts; only the first was checked, so
                             splitting a rule file could not fail here
  docs/images/README.md      "five dashboards" in a file not in PROSE
                             at all, despite being present-tense

Counts are now matched spelled out as well as in digits, which is what
made the last two invisible. The word list stops at twenty to keep the
alternation short, and is ordered longest-first so "seven" cannot match
inside "seventeen" and leave the rest of the pattern to fail.

Adding docs/images/README.md to PROSE also brings it under the other
five assertions, and it passes them unchanged.

Each new assertion was mutation-tested in both directions, following the
convention b316399 set. Changing the prose fails:

  README.md:158 claims 12 LogQL rules; the repository has 13
  docs/observability.md:111 claims seven Prometheus rule files; the repository has 6
  docs/images/README.md:84 claims six dashboards; the repository has 5

and so does changing the configs the prose describes — a seventh rule
file alone trips the file count, and a sixth dashboard trips both lines
of docs/images/README.md that were previously unguarded.

Also adds `make check-docs`. It was the only check reachable from
validate.sh and CI but not the Makefile, which is the asymmetry #68
recorded for snmp-targets.sh and the same commit fixed there.

Refs #72, #68

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 4db6db1 into main Aug 31, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/stale-alert-panel-counts-853424 branch August 31, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant