feat(debug): rolling debug-history snapshot buffer with automation hooks

[chalfontchubby]

Summary

Fixes #4417.

create_debug_yaml() (what bug reporters attach so a maintainer can replay their exact plan computation via unit_test.py --debug_file --redo) only ever wrote when switch.predbat_debug_enable was already on, on every ~5-minute cycle with no pruning at all - unbounded disk growth if left on, and no history to look back at for anyone who didn't already have it on before something went wrong (originally raised in #1745, never actually built).

Adds a rolling buffer of debug snapshots, independent of that switch, so there's always some recent history to look back at:

• debug_history.py (new) - capture/list/load/prune snapshots, mirroring the existing annual_store.py ring-buffer pattern and going through the Storage component rather than the filesystem directly. Snapshots are saved with format="text" (the payload is already-rendered YAML text from create_debug_yaml(write_file=False), not a Python object).
• Four new config items: switch.predbat_debug_history_enable (default on) turns the rolling capture off entirely; debug_history_count (default 15, minimum 1) and debug_history_interval (default 3h - 15x3h matches the History tab's worst-case "yesterday + today so far" coverage) size the buffer; debug_history_force_capture is a self-resetting switch so an automation that notices something worth investigating can trigger an immediate, precisely-timed capture rather than waiting on the routine interval - still honoured even when the rolling capture is disabled.
• predbat.py: _capture_debug_history(), throttled in-memory, only advances its throttle clock or resets the force-capture switch on a genuine successful capture - a failed attempt (or Storage being unavailable) retries next cycle instead of silently deferring a full interval.
• switch.predbat_debug_enable (the existing raw per-cycle debug.yaml writer) now auto-disables itself after DEBUG_ENABLE_MAX_HOURS (2h) if left on, bounding the unbounded-disk-growth problem for that path too without removing it outright - it's still useful at finer resolution than the rolling buffer's hour-plus floor for someone actively watching a live issue.
• web.py: /debug_history_list and /debug_history_download routes (reusing the existing html_file() attachment-serving helper), plus a single Download all (.tgz) link on the dashboard's Debug panel that bundles every retained snapshot.
• web_helper.py: a Debug column on the plan's History/Yesterday view only (its rows have a real past timestamp to match a snapshot against - the live Plan view is mostly future predictions with nothing to match). A link appears when a row lands on the exact floored capture slot for a snapshot; fetched and matched client-side by wall-clock time.
• Docs - customisation.md ("Debug history" subsection, plus a note on debug_enable's new auto-disable), web-interface.md (the download link), predbat-plan-card.md (the Debug column).

Doesn't touch car charging forecasting or historical cost reporting - this is purely a debugging/diagnostic aid, no effect on planning behaviour.

Test plan

• test_debug_history.py - capture/list/load round-trip, pruning to a configured count on both Storage eviction paths, same-minute dedup, max_age pruning, corrupt-index handling, latest/falsy-id resolution, and a real-StorageLocalFiles round-trip of genuine YAML-syntax text (guards the format="text" vs "yaml" trap directly)
• test_debug_history_capture.py - interval throttling, the enable switch disabling routine capture, force-capture bypassing both the count and interval, the switch resetting itself only on success, graceful handling with no storage component available
• test_debug_enable_auto_scope.py - the 2-hour auto-disable behaviour
• test_web_debug_history_routes.py - all three new routes against a real StorageLocalFiles backend, including atomic latest-snapshot resolve+load and reflected-id escaping
• test_debug_history_client_js.py - structural coverage for the client-side JS (findNearestDebugSnapshot/loadDebugHistoryData/Debug column rendering), following test_plan_why_reason.py's precedent for testing embedded JS
• ./run_all --quick - full suite passes
• ./run_pre_commit - clean

🤖 Generated with Claude Code

[chalfontchubby]

Adds the ability to download debug files for a defined subset of points in the history without the ever expanding set of debug files enabled with "debug". This screenshot covers a period with multiple restarts, so the snapshot files are not at the expected regular intervals.

A follow on branch adds a new chart showing how these "historical plans" evolve over time (the planned soc overlaid for each historical snapshot) so it is possible to find out "what caused that weird thing I see in the history" and "why did the plan suddenly do something different"

[springfall2008]

Review

Nice feature — went through the diff in detail. A few correctness issues worth fixing before merge, plus some lower-priority cleanup.

Correctness bugs

1. predbat.py _capture_debug_history() — self.debug_history_last_capture = self.now_utc runs unconditionally after the try/except. A failed capture (exception caught and logged as a "Warning") still resets the throttle clock, so the next attempt is silently deferred for a full debug_history_interval (default 3h) instead of retrying next cycle.
2. Same function, storage-unavailable path — if not storage: return exits before that line, so the throttle timestamp never advances at all. If Storage is ever down, every ~5-minute cycle re-enters and re-logs the warning indefinitely.
3. Force-capture switch resets before the outcome is known — expose_config("debug_history_force_capture", False) fires immediately on trigger, before storage is checked or the capture is attempted. docs/customisation.md says "Predbat resets the switch back off itself once the snapshot has been taken" — the code resets it on trigger, not completion. An automation polling the switch for success has no reliable signal if the capture actually failed.
4. web.py html_debug_history_download, id=latest case — load_snapshot(storage, "latest") resolves "latest" internally and discards the resolved id; the route then calls list_snapshots() a second, independent time just to rebuild the filename. A capture landing between the two calls can serve one snapshot's bytes under another snapshot's filename.
5. debug_history.py capture_snapshot() max_age pruning — the try/except (KeyError, ValueError, TypeError) only wraps the fromisoformat parse, not the later existing_time < cutoff comparison. A naive (non-tz-aware) stored timestamp would raise an uncaught TypeError, aborting the index write and orphaning the just-captured snapshot in storage. Low likelihood today since now_utc is always tz-aware, but the guard doesn't cover what its comment claims it does.
6. web.py — "Snapshot {} not found".format(snapshot_id) reflects the raw ?id= query param unescaped into a text/html response. Reflected-XSS shape on a newly-added, unauthenticated route (this mirrors an existing pattern in html_file(), but it's new attack surface worth escaping while we're here).
7. Same-slot force-captures silently overwrite each other — the snapshot id is floored to the plan-slot grid, so firing debug_history_force_capture twice within one slot produces the same id and the second storage.save() silently clobbers the first — no log, no dedup/discard path.

Docs no longer match shipped behavior

8. docs/web-interface.md and docs/customisation.md describe a dropdown snapshot picker on the dashboard Debug panel. The merged code only adds one link, Download all (.tgz) — an earlier commit in this PR replaced the dropdown, docs weren't updated to match.
9. docs/predbat-plan-card.md still says snapshots are matched "nearest" to a slot and labelled by "how many snapshots ago" — the shipped JS does exact floored-slot matching and labels with absolute time instead.

Efficiency

10. create_debug_yaml() runs twice per cycle when debug_enable is on and a capture is due — update_pred() calls it directly, then _capture_debug_history() calls it again internally.
11. html_debug_history_download's id=latest path does two full index round-trips (see solcast entity nomenclature  #4).
12. load_all_snapshots awaits snapshot bodies sequentially instead of asyncio.gather (already used elsewhere, e.g. annual_job.py).
13. The bulk .tgz route buffers every snapshot body plus the full archive in memory rather than streaming — worth a thought given Predbat commonly runs on Pi-class hardware.
14. findNearestDebugSnapshot does an O(rows × snapshots) linear scan with repeated Date parsing per row, on every 5s poll of the Yesterday view — could build a Map once when debugHistoryData is fetched.
15. The plan table renders once synchronously, then again after loadDebugHistoryData() resolves — a visible flicker on first entry to the History/Yesterday view.

Reuse / duplication

16. web.py's new _storage() helper duplicates ComponentBase.storage, which WebInterface already inherits — could just call self.storage.
17. debug_history.py's ring-buffer (list/discard/prune) logic duplicates annual_store.py's equivalent rather than sharing it, and has already diverged (dedup + max_age only added to one side).
18. The plan-slot flooring math in _capture_debug_history() re-derives from scratch what output.py already computes via self.minutes_now.

Simplification

19. annotate_steps_back()/steps_back is computed and shipped on every /debug_history_list response, but the final JS only reads .id/.timestamp — looks like a leftover from an earlier design (before the move from "N steps back" labels to absolute time).
20. Three near-identical prune loops (same-minute dedup, max_age, max_count) could collapse into one parametrized helper.

Test coverage

Per this repo's usual bar, a couple of gaps:

21. The three new web.py routes and _storage() have no test coverage — tests/test_web_annual.py has an established pattern for this (build a FakeRequest, call the handler directly) that isn't applied here.
22. The new client-side JS (findNearestDebugSnapshot, loadDebugHistoryData, the Debug column rendering) is untested — tests/test_plan_why_reason.py has a precedent for testing this kind of embedded JS via a Python mirror of the function.

Minor

23. Dead .cspell dictionary entry (onfocus) left over from the removed dropdown markup.
24. Same-minute dedup keys on local-time strings, so a UK DST fall-back's repeated HH:MM could in theory collide and silently drop a real, distinct capture.

Most worth fixing before merge: #1–#4 and #6 (correctness/security), #8–#9 (docs actively mislead users right now), and #21–#22 (test coverage). The rest is good-to-have cleanup, happy to see it split into a follow-up if you'd rather keep this PR focused.

[springfall2008]

Suggestion: split debug_enable's file-write behavior from this feature, and give the rotating buffer its own switch

Two related follow-ups worth doing in this same PR while we're touching this area:

1. debug_enable still causes unbounded disk growth — the original problem isn't actually fixed yet

predbat.py's if self.debug_enable: self.create_debug_yaml() (still present, right next to the new _capture_debug_history() call) writes a brand-new predbat_debug_HH_MM_SS.yaml straight to config_root via raw os.makedirs/open() every ~5-minute cycle, with no cap and no pruning, whenever a user leaves the switch on. That's exactly the "unbounded disk growth if left on" issue this PR's own commit message cites as the motivation for the rotating buffer — but this PR adds the new bounded buffer alongside it rather than replacing it, so the original bug ships unchanged.

Suggest removing that call site entirely. Nothing is lost:

• The /debug_yaml route already serves the same content on demand without writing to disk (write_file=False).
• The new rotating buffer now covers "what was Predbat doing recently" automatically.

debug_enable's other ~40 usages (verbose logging in plan.py/execute.py/octopus.py, and the cache/kernel-reuse bypass in prediction.py/prediction_kernel.py for more accurate step-by-step debugging) are a separate concern and should stay as-is — only the disk-write coupling should go.

2. Give the rotating buffer its own on/off switch, rather than overloading debug_history_count == 0

Right now the feature's only off-switch is setting the debug_history_count input_number to 0. That's workable but inconsistent with how this codebase normally does on/off — debug_enable and debug_history_force_capture are both real switch CONFIG_ITEMS. Suggest adding switch.predbat_debug_history_enable (defaulting either way, your call) so "is this feature on" and "how many snapshots do I keep" are independent settings, matching the existing pattern instead of a magic-zero convention.

[chalfontchubby]

Thanks for the thorough review - went through it in priority order. Everything flagged as "most worth fixing before merge" is addressed, split into separate commits so each is reviewable on its own:

Correctness (items 1-4, 6)

• 50d33564 - debug_enable still caused unbounded disk growth even after this PR - the rotating buffer shipped alongside the original raw write, not in place of it. Rather than remove the write outright (it, and the verbose logging/kernel-bypass path it also gates, are genuinely useful at finer resolution than the history buffer's 1-hour floor for someone actively watching a live issue), it now auto-disables after 2 hours if left on.
• f1de7ed0 - items 1-3: debug_history_last_capture and the force-capture switch now only reset on a genuine successful capture, not unconditionally. A failed attempt no longer silently defers the next routine retry a full interval, the storage-unavailable warning is throttled instead of re-logging every ~5 minutes, and a failed forced capture leaves the switch on to retry rather than resetting before the snapshot was actually taken.
• 50706e09 - items 4 and 6: added resolve_and_load_snapshot() so id=latest resolves and loads in one call instead of two independent ones (closing the race where a capture landing in between could serve mismatched bytes/filename), and escaped the reflected ?id= query param in the 404 body.

Docs (items 8-9)

• 60b6af40 - updated web-interface.md, predbat-plan-card.md, customisation.md to describe the single "Download all (.tgz)" link and exact floored-slot matching that actually ships, not the earlier dropdown/nearest-match design.

Tests (items 21-22)

• 50706e09 also added test_web_debug_history_routes.py (all three routes, including the item 4/6 fixes, against a real StorageLocalFiles backend).
• 1134c514 added test_debug_history_client_js.py for findNearestDebugSnapshot/loadDebugHistoryData/the Debug column - following test_plan_why_reason.py's existing precedent for this kind of embedded-JS test (asserting on the JS source structure, since there's no JS engine in this suite).

Design suggestions in your second comment

• Went with your switch.predbat_debug_history_enable suggestion (7e790fc2) - also raised debug_history_count's minimum from 0 to 1, so it can no longer double as a second, conflicting off-switch.
• Kept the raw debug_enable disk write rather than removing it outright, for the reason in 50d33564 above - open to discussing further if you'd still rather see it gone entirely.

Full suite + pre-commit clean after every commit. The remaining items (#5, #7, #10-20, #23-24) are left as follow-up per your own note that they're good-to-have cleanup rather than blocking - happy to pick any of them up next if you'd like them in this PR instead.

[chalfontchubby]

@springfall2008 i think most/all
Your concerns have been addressed- could you take another look.

[springfall2008]

Reviewed this — solid implementation, nice test coverage on the storage/pruning paths and the JS structure. Didn't find any functional bugs in the capture/prune/route logic (traced the eviction fallback, the format="text" vs "yaml" handling, and the ?id= download route for injection risk — all fine). Two things worth resolving before merge:

1. PR description vs. shipped UI don't match. The description says web.py adds "a dropdown on the dashboard's Debug panel" for browsing snapshots, but the diff only adds a single "Download all (.tgz)" link to the status table — there's no per-snapshot picker on the dashboard itself. The only ways to grab one specific older snapshot are: know its ?id= already, use ?id=latest, or find it via the plan's History/Yesterday view's Debug column (which only shows a link when a plan row lands on the exact floored capture slot). Was the dropdown simplified away in an earlier review round and the description just never got updated, or is this a gap worth adding?

2. The new debug_enable auto-disable-after-2h behaviour isn't documented. docs/customisation.md already has a section on switch.predbat_debug_enable (with a user-recipe HA automation that manually turns it off after 10 minutes for exactly this "left on by accident" reason). This PR adds a built-in DEBUG_ENABLE_MAX_HOURS auto-disable, which is a real user-visible behaviour change for anyone running a longer debug session, but only the new "Debug history" doc subsection was added — the existing debug_enable writeup wasn't touched. Worth a one-line mention there so this doesn't surprise anyone.

Minor nit: in test_debug_history.py, the same-minute-dedup test's rationale comment ("this used to reach the UI as two identically-named chart series and broke 'Deselect all'") looks like it was copied from annual_store.py's equivalent test — debug-history snapshots aren't rendered as chart series anywhere in this PR, so the comment's "why" doesn't actually describe this feature. Not wrong, just a confusing pointer for a future reader.

[chalfontchubby]

I've been nagging Claude to keep the pr description valid on an update, but it's on me that I didn't notice we had drifted.

Deselect all is leakage from a follow on patch that adds a chart plotting Soc predictions for each of the debug snapshots which has helped diagnose drift between evolving plans. That is quite a nerdy chart though (not the only one in predbat perhaps).

[chalfontchubby]

I'll update the docs in a bit.

[chalfontchubby]

Docs updated - customisation.md now mentions the 2-hour debug_enable auto-disable next to the existing writeup, and the same-minute-dedup test comment in test_debug_history.py no longer references the unrelated "Deselect all" chart-series issue.

Also rebased onto current main to clear the merge conflicts - branch is clean and CI's running now.