fix: Use calibration/validation splits correctly, fix walk-forward bl…

[Samarthuday]

…ock bug

Two genuine methodology issues survived the prior integration-fix branch, both confirmed by reading the actual code (not just the bug report) before changing anything:

1. train_model.py fit the probability calibrator AND selected the alert threshold from the same calibration split -- the separate validation dataframe from temporal_split was computed and logged but never actually used for anything. Fixed: calibrator now fits only on calibration; threshold selection and the calibration-quality diagnostic (Brier/log-loss, calibration_curve.png) now use the true held-out validation split. The baseline logistic model's calibrator is likewise fit on calibration (matching the main model) rather than on validation.

   feature_ablation.py had the same unused validation parameter, but with no natural role there (ablation only measures top-K ranking quality, no threshold is chosen) -- removed rather than forcing an artificial use.

2. walk_forward_backtest.py indexed into the raw unique-timestamp array using literal offsets (train_end = 4 + window*2, etc). On a dataset with 7.75M unique timestamps this meant every window only ever used the first ~14 timestamps of the entire dataset -- which is why it previously failed with "no valid walk-forward windows were available" on the real SAML-D run. Fixed: timestamps are now split into 14 contiguous chronological blocks via np.array_split first, and the same expanding-window indices are applied to block boundaries instead. Verified against the real 9.5M-row dataset: now produces 4 real windows spanning the full timeline (PR-AUC 0.938 -> 0.990 as the training window expands), instead of raising. Extracted the boundary logic into window_boundaries() and added 3 regression tests (tests/test_walk_forward.py) so this can't silently regress back to a same-bug-different-shape variant.

Also, two smaller items from the same review:

3. unseen_entity_evaluation.py only reported per-subgroup top-K metrics (each subgroup gets its own top 0.5%). Added metrics_at_threshold() to src/evaluation/metrics.py and a second table applying the artifact's actual fixed decision_threshold to both subgroups -- answers "what does the current production policy do to unseen accounts," distinct from "what's the best possible ranking within this subgroup alone."

4. run_experiments.py never ran generate_report_figures.py, so the documented single-command workflow didn't actually produce the figures the dashboard/README reference. Wired it in, and added the unseen-entity experiment behind an explicit --generalization flag rather than forcing the research extension into every run.

generate_report_figures.py and unseen_entity_evaluation.py now also accept --features/--artifact so they honor custom paths passed to run_experiments.py instead of silently falling back to defaults.

Status: all 63 existing tests + 3 new tests pass, ruff clean. Walk-forward fix independently verified against the real dataset (above). The calibration/validation split fix is verified by tests and only changes the saved artifact's decision_threshold/validation_probability_quantiles and calibration diagnostics (Brier/log-loss/calibration_curve.png) -- it cannot change test-set PR-AUC/ROC-AUC, since those are computed from the test split independently of where the threshold was chosen. Retraining to refresh the committed artifact and docs/assets/ diagnostics with the corrected methodology is a separate follow-up, not yet done.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.