feat(forecast): use learned PV and load as primary forecasts

[frahlg]

Accepted text proposal

Maintainer-requested forecasting work in Codex: learn PV without required panel setup, use the new Rust pipeline as the first forecast source, retain the old forecast as a shadow, and publish a beta after verification.

Paired private model PR: https://github.com/srcfl/energyplan/pull/3. Rust source, tests and build tools stay there; FTW receives compiled workers and public contracts. The Windows build dependency #1088 has landed.

What changed

The planner now selects Energyplan PV and household-load predictions first, including valid cold-start, learning and zero values. The previous forecast runs as a frozen legacy_shadow and supplies an explicit fallback for each signal when the new prediction is unknown, invalid or unavailable.

• Resolve the complete forecast horizon once before planning, using frozen model state, weather and occupancy. A two-second worker deadline and replan cancellation bound the call outside service and dispatch locks. Preserve price data, slot times and physical limits.
• Predict the remaining part of the current quarter too, so the first control interval can use Rust. Record its actual prediction start and exclude partial intervals from accuracy, calibration and cumulative-error scores.
• Apply risk to the selected forecast. Use matching empirical joint-net errors when available, otherwise the selected model's provisional bounds or an explicit prior. Keep the point forecast separate from the actual risk-adjusted solver inputs.
• Save champion, planning, legacy_shadow, raw energyplan and baselines from one capture. Record PV/load source per point. The read-only evaluator defaults to the actual primary versus the legacy shadow from that same capture.
• Correct provider interval timing, Forecast.Solar direction and energy aggregation. Reject stale or missing weather and prevent a previous weather configuration from repopulating the cache.
• Train from coherent, fresh measurements; retain true OCPP power-sample times; qualify household and PV evidence independently; exclude unresolved commanded PV curtailment. Bind learning to live hardware identity, measurement configuration and location.
• Preserve compatible learning through restarts and program upgrades. Persist new Rust state atomically before exposing it to the planner. Keep the archive, queues, model sizes and retained history bounded.

Why

The earlier path could train on incomplete power balances, shift radiation by an hour, and turn a guessed PV rating or grid fuse into a forecast limit. Repeated samples also earned trust too quickly. Those failures now have regressions.

The new default follows the maintainer's requested beta cutover. A valid Rust prediction is used immediately; measured field accuracy remains to be established during beta operation.

Boundaries and safety

Core still validates every plan and command. Stale meter data stops dispatch. Forecast failures use the frozen legacy fallback; they cannot issue hardware commands. A canceled or superseded forecast cannot publish a plan. The legacy shadow cannot change active forecast inputs or trigger its own model-drift replans.

Energyplan 0.2.1 adds faster adaptation to sustained valid low PV, preserves the learned normal curve, and accepts predictions for future parts of a quarter. Observation updates still require complete quarters. Forecast and optimizer protocols remain v1. The new reader preserves compatible 0.2.0 state; replay of new model snapshots requires the recorded worker version.

Opaque model state and all weather/occupancy inputs are archived for replay. Historical rows without measurement qualification do not become training truth. The curtailment receipt writes asynchronously to keep storage off dispatch; a crash before that write completes remains a limitation. Unknown release behavior stays excluded from PV training.

Open overlap #734 remains intact. A merge simulation found one text conflict in the forecast log/return and a callback-start ordering issue in main.go. Its later rebase must retain the captured provider/generation checks, wire its calibration callback before service start, and apply STRÅNG calibration only to its weather/legacy prior. The Rust model consumes raw weather. This PR ships without importing #734's map, UI or roof stack; that PR keeps its existing review work.

Verification

Verified Core head: 9475683c4e2c133aef7982af447c11cd6b3b7c08, based on master 821a5f660bae24ac0bc9cea071699b5ce7d18e0d.

• Energyplan 0.2.1 source: 820383e99ff21ae1a1428ebbf97bfaa9768e1cd5. Private PR docs: add device identity reference #3 merged as 5bd7ed186558464d8f55352e8f0c5d0e47d429e6 after green CI; source and merge have the same tree.
• Public bundle verifier: all 20 distribution files match their checksums and sizes; three target architectures and the actual Mac ARM64 worker handshake pass.
• make verify-all: Go tests, vet, build, native integrations, release/container/migration checks, and Linux ARM64, Linux AMD64 and Windows AMD64 cross-builds pass.
• Race checks pass for MPC, the forecast client, forecasting, the host and the read-only evaluator with both worker environment variables set to the actual 0.2.1 bundle.
• Native regressions pass for current-quarter selection, replay after a new observation, occupancy replay, state restore, and primary-versus-risk solver/archive inputs.
• The private worker passes 95 tests, v1-state migration, 1,161 update/restore cycles, bounded-state checks and a synthetic causal low-output regression. These checks do not establish accuracy on a real site.
• No box installation or physical-site forecast validation has run as part of this beta cut. The maintainer requested publication before that trial.

To compare forecasts from a box backup, run from go/:

go run ./cmd/ftw-forecast-evaluate -state /path/to/state.db

This reads SQLite without migrations and prints JSON. -primary and -reference select other archived series. The command is a source tool, not a new release executable.

Checklist

• The diff implements one accepted scope and does not add follow-on work.
• I checked open pull requests that touch the same files.
• Tests cover the changed behaviour and its failure path.
• No web/UI view changed.
• A Changeset is included.
• Every commit has a DCO sign-off.

---

Note

Medium Risk
Changes replanning inputs, measurement qualification, and PV curtailment labeling on the path to dispatch, though invalid worker output falls back to legacy and control safety boundaries are preserved.

Overview
Makes the Energyplan worker the planner’s first PV and household-load forecast, with per-signal fallback to a frozen legacy shadow when predictions are missing, invalid, or the worker times out. Replan captures weather, occupancy, and model state once, calls the worker outside control locks, and archives champion, planning, legacy_shadow, raw energyplan, and baselines for causal comparison.

Tightens what counts as training and history truth: complete fresh 15‑minute balances, independent PV vs load evidence, stricter household history when DER readings are stale, and PV curtailment labeling via a non-blocking wrapper keyed on stable device identity and reviewed driver release semantics. Learned Rust state persists atomically in SQLite; site learning vs evaluation revisions bind hardware identity, measurement topology, worker bytes, and pipeline policy.

Adds ftw-forecast-evaluate (read-only JSON scores from the forecast archive), extends native CI/Makefile tests for energyforecast and forecast host paths, and updates architecture docs so the optimizer contract covers forecasts as well as plans.

^{Reviewed by Cursor Bugbot for commit 9475683. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T11:49:16.172292Z	9475683	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9475683. Configure here.}

[cursor]

Nil PV service crashes snapshot

High Severity

Snapshot always calls f.pv.ForecastSnapshot() even though the same function later treats f.pv as optional. Sites without a weather-backed PV service (provider unset or none) leave pvSvc nil, so the first replan can panic instead of falling back to load-only or legacy planning.

 

^{Reviewed by Cursor Bugbot for commit 9475683. Configure here.}

[cursor]

Deadline discards valid primary forecast

Medium Severity

After Predict returns a valid result, Resolve still replaces it with the legacy forecast if workCtx has already expired. A worker that finishes at the two-second bound is treated as a failure, so the planner silently drops Energyplan on the tight deadline that already wraps the call.

 

^{Reviewed by Cursor Bugbot for commit 9475683. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9475683c4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Capture the freshness cutoff at persistence time

During a normal control tick, nowMs is captured near the beginning of the tick, but this history snapshot runs only after dispatch and other processing. If any concurrent driver poll updates telemetry in that interval, ForecastMeasurement sees its UpdatedAt as later than this reconstructed now and rejects the fresh reading as future, causing the entire history point and observed-consumer ledger row to be omitted. Use the current time as the measurement/freshness cutoff while retaining nowMs only as the history bucket timestamp.

Useful? React with 👍 / 👎.

[cursor]

Left a non-blocking comment; not approved. Cursor Bugbot reported two unresolved findings (including a high-severity nil PV snapshot panic) and the check ended skipped, so this needs human review. No reviewers assigned — the only matching CODEOWNER is the PR author.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}