fix(ci): guard the counted claims the checker was still missing - #195
Merged
Conversation
scripts/check_docs.py closed #72 by deriving the counts from the configs, but it only ever looked for the phrasings someone happened to think of. Three present-tense claims sat outside it, all correct and all able to go stale silently — the failure #72 is about, surviving inside its own fix. README.md:158 "13 LogQL rules" — the Loki pattern is `log-based`, so this matched nothing, one line away from a checked "13 log-based" docs/observability.md:111 "39 rules across six files" states two counts; only the first was checked, so splitting a rule file could not fail here docs/images/README.md "five dashboards" in a file not in PROSE at all, despite being present-tense Counts are now matched spelled out as well as in digits, which is what made the last two invisible. The word list stops at twenty to keep the alternation short, and is ordered longest-first so "seven" cannot match inside "seventeen" and leave the rest of the pattern to fail. Adding docs/images/README.md to PROSE also brings it under the other five assertions, and it passes them unchanged. Each new assertion was mutation-tested in both directions, following the convention b316399 set. Changing the prose fails: README.md:158 claims 12 LogQL rules; the repository has 13 docs/observability.md:111 claims seven Prometheus rule files; the repository has 6 docs/images/README.md:84 claims six dashboards; the repository has 5 and so does changing the configs the prose describes — a seventh rule file alone trips the file count, and a sixth dashboard trips both lines of docs/images/README.md that were previously unguarded. Also adds `make check-docs`. It was the only check reachable from validate.sh and CI but not the Makefile, which is the asymmetry #68 recorded for snmp-targets.sh and the same commit fixed there. Refs #72, #68 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the close-out of #72. Two gaps found while verifying that issue, both recorded in its closing comment.
Neither was a stale count.
scripts/check_docs.pyalready derives the counts from the configs, and every number in the repository is currently correct. These are holes in the guard itself — claims it could not see, which would therefore drift silently. That is the failure #72 documents, surviving inside its own fix.Three unguarded claims
README.md:15813 LogQL ruleslog-based. One line from a checked "13 log-based", and matched nothing.docs/observability.md:11139 rules across six filesdocs/images/README.md:50,84five dashboardsPROSEat all.The last two are spelled out, which is what made them invisible — the checker only matched digits. Counts are now matched either way. The word list stops at twenty to keep the alternation short (not a claim about the ceiling) and is ordered longest-first so
sevencannot match insideseventeenand leave the rest of the pattern to fail.Bringing
docs/images/README.mdintoPROSEalso subjects it to the other five assertions. It passes them unchanged, and its dashboard claims are accurate today: 5 dashboard JSONs, 4 PNGs,homelab-logsexcluded on purpose incapture-screenshots.sh:56-60.Mutation tests
Each new assertion was tested in both directions, following the convention
b316399set.Changing the prose fails:
Changing the configs the prose describes fails too — which is the direction that actually matters, because it is how the Suricata, firewall-log and UPS self-test rules went stale in the first place. A seventh rule file carrying no new alerts trips the file count on its own:
and a sixth dashboard trips both previously-unguarded lines:
Adding one real alert rule produces ten failures across six files, including
docs/runbooks/deploy-stack.md:107— the deploy-runbook step that made #72 worth filing.make check-docscheck_docs.pywas the only check reachable fromvalidate.shand CI but not the Makefile. That is the same asymmetry #68 recorded forsnmp-targets.sh, whichb316399fixed on the CI side; this closes it on the Makefile side. Placed beforecheck-dashboardsto match the order CI runs them in.Verification
./scripts/validate.shpasses end to end, including gitleaks over the full history and the lint that assertsci.yml, the Makefile andvalidate.shall delegate to the same scripts.No document or config changed — only the checker and the Makefile.
Refs #72, #68
Generated by Claude Code