fix(ml): resolve NaN training loss on startup and preserve database zero entries

[rholligan]

Summary

Fixes ML load forecaster startup training aborting with huber_loss=nan at Epoch 1 when historical database records contain missing elements (NaN/None) or valid 0.0 readings.

Why

• np.maximum(np.nan, min_std) preserves NaN in standard deviations, causing division by zero/NaN in normalization and corrupting forward passes.
• dict_to_array initialized arrays with 0.0, causing array_to_dict to drop valid 0.0 load/PV readings on reload.
• _do_training() triggered duplicate consecutive curriculum passes, and intermediate passes failed to retain model initialization state.

What Changed

• apps/predbat/load_predictor.py: Hardened feature/target normalization against NaN/Inf, added gradient clipping ([-10.0, 10.0]), finite weight checks, and preserved curriculum initialization state.
• apps/predbat/load_ml_component.py: Initialized history arrays with np.nan to preserve 0.0 data points, and removed redundant duplicate training call in _do_training().
• apps/predbat/tests/test_load_ml.py: Added _test_nan_inf_robustness, _test_database_zero_preservation, and _test_curriculum_90day_intermediate_passes.
• apps/predbat/unit_test.py: Reconfigured console stdout/stderr to UTF-8 for cross-platform test runner stability.

Test Plan

• Added _test_nan_inf_robustness, _test_database_zero_preservation, and _test_curriculum_90day_intermediate_passes in tests/test_load_ml.py (32/32 tests passed).
• Full ./run_all --quick suite passes.
• pre-commit (black, ruff, cspell, markdownlint) clean.

✦ Developed with Google Antigravity

[Copilot]

Pull request overview

This PR targets robustness and correctness in Predbat’s ML load forecaster by preventing NaN/Inf contamination during training (especially on startup with imperfect historical data) and by ensuring persisted history round-trips don’t silently drop legitimate 0.0 readings.

Changes:

• Hardened LoadPredictor normalisation/training math against NaN/Inf and added safety measures (gradient clipping + finite-value clamps).
• Changed ML history persistence to use NaN as the “missing” sentinel so 0.0 values survive save/load, and removed redundant duplicate curriculum training invocation.
• Expanded ML tests to cover NaN/Inf robustness, DB zero preservation, and longer curriculum intermediate-pass behaviour; plus a few cross-platform test runner improvements.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
apps/predbat/load_predictor.py	Adds NaN/Inf sanitisation in normalisation paths and training safety clamps to prevent NaN loss at startup.
apps/predbat/load_ml_component.py	Changes DB save/load encoding to preserve explicit zeros and refactors curriculum training invocation.
apps/predbat/tests/test_load_ml.py	Adds regression tests for NaN/Inf robustness, DB zero preservation, and 90-day curriculum intermediate passes.
apps/predbat/unit_test.py	Forces UTF-8 stdout/stderr in the test runner for cross-platform output stability.
apps/predbat/tests/test_sunsynk_api.py	Adds a SIGALRM-less fallback hard timeout implementation for platforms without signal.SIGALRM.
apps/predbat/tests/test_plan_why_reason.py	Opens source files with explicit UTF-8 encoding for portability.
apps/predbat/tests/test_ml_training_perf.py	Makes resource usage optional so the perf harness can run on platforms without it.
apps/predbat/tests/test_download.py	Normalises test output strings and makes the SHA1 fixture use binary mode for consistent LF handling.
.cspell/custom-dictionary-workspace.txt	Adds nanmean/nanstd to the spellchecker dictionary.

Suppressed comments (2)

apps/predbat/load_ml_component.py:840

• dict_to_array() writes arr[idx] = float(value) without guarding against None/non-numeric values. If any history dict contains None (or an unparsable value), save_database_history() will raise TypeError/ValueError and skip persisting the database entirely. Given the ML pipeline explicitly tolerates None/NaN in inputs, the DB save path should treat these as missing and leave the default NaN sentinel in place.

                for minute, value in data_dict.items():
                    # Only persist historical data (non-negative integer keys)
                    if isinstance(minute, int) and minute >= 0:
                        idx = minute // PREDICT_STEP
                        if 0 <= idx < max_steps:

apps/predbat/load_ml_component.py:906

• array_to_dict() decides whether to preserve 0.0 values based on whether the loaded array contains any NaNs. This heuristic fails when a NaN-sparse file happens to have no NaNs (fully-populated history) and will revert to the legacy "0.0 means missing" logic, dropping valid zeros. Use the explicit metadata flag (and keep a fallback heuristic for older files) to choose decoding logic.

                has_nans = np.isnan(arr).any()
                for i in range(len(arr)):
                    val = float(arr[i])
                    if has_nans:
                        if not np.isnan(val) and np.isfinite(val):

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rholligan]

Addressed all automated review findings in commit 4ff478a:

• load_ml_component.py: Added try/except and None guard in dict_to_array(), and added "nan_sentinel": True metadata flag so fully-populated history files preserve explicit 0.0 values.
• load_predictor.py: Added posinf=0.0, neginf=0.0 to post-update safety clamp in _adam_update().
• test_load_ml.py: Extended regression tests covering fully-populated 0.0 preservation, invalid/unparsable sensor inputs during DB save, and +/-Inf weight clamping.