chore: detect silent launchd gaps in check_sensor_health.py - #572
Open
essentialbit wants to merge 2 commits into
Open
chore: detect silent launchd gaps in check_sensor_health.py#572essentialbit wants to merge 2 commits into
essentialbit wants to merge 2 commits into
Conversation
The watchdog only logs when it actually fires, so a launchd job that stops firing entirely (unloaded, machine asleep past its wake window, etc.) leaves no trace distinguishable from "nothing needed reporting" - the existing streak/gap_hours output requires manual arithmetic to notice. Caught live: 2026-08-04 19:29 -> 2026-08-15 23:54, a 268.4h gap with zero watchdog.log activity, sitting right after a 22-cycle weekly-limit failure streak that made the whole outage look explained by the limit alone. Scans every watchdog.log line's timestamp (not just cycle-finish markers) so start/skip lines count too, and flags any gap >= 2h (launchd's StartInterval is 3600s) as a distinct SILENT_GAP finding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check_sensor_health.py'sgap_hours(time since lastrc=0cycle) correctly captures elapsed wall-clock time, but nothing explicitly flags when that gap is wider than the failure streak alone would explain — i.e. launchd itself stopped firing, distinct from "fired and errored" (rc!=0).Caught live this cycle: the sensor logged 22 consecutive
weekly_limitfailures from 2026-08-04 00:20 to 19:29 (~19h), then zero watchdog.log activity for 268.4h (2026-08-04 19:29 → 2026-08-15 23:54) before this cycle fired again.launchctl printshowedruns = 1on the job, confirming it had just (re)started — the machine itself had been up continuously since an 2026-08-08 reboot, so the job was silent for ~7 days despite the host being available. At a glance the existing output (22x weekly_limit, 290.7h since last success) reads as "one long weekly-limit outage," masking the much larger silent gap.Changes
find_silent_gaps(): scans every watchdog.log line's leading timestamp (start/skip/finish all count, not just cycle-finish markers), flags consecutive-line gaps >= a threshold (default 2h, vs. launchd's 3600sStartInterval).SILENT_GAPsection in the report output, separate from the existing streak/verdict logic, so a launchd outage is never absorbed into (or hidden behind) an unrelated failure-streak explanation.--gap-threshold-hours/--max-gapsCLI flags.No behavior change to the existing
verdict/streak logic — purely additive.Test plan
~/.claude/fred-sensor/watchdog.log— correctly isolates the 268.4h gap as a distinct finding from the 22xweekly_limitstreak.PYTHONPATH=. python3 -c "from main import app"— no import regression.ast.parse()on the modified file.scripts/*.pyin this repo); diagnostic/read-only, single-file, self-directed-tooling class per project policy — no consensus/outcome-recording gate applies.🤖 Generated with Claude Code