Do published probabilistic forecasts hold up when you score them?
Institutions publish probabilistic forecasts every day — precipitation probability, air-quality categories, day-ahead electricity prices. Most are never independently verified, for a mundane reason: verification requires capturing the forecast and the outcome as they are published. Forecasts get revised. Archives get overwritten in place. By the time you want to check, the original claim is often gone.
This repository self-collects both sides in three domains and scores them with one shared toolkit: proper scoring rules, Murphy decomposition, CORP reliability, bootstrap CIs.
One method, three domains. The interesting part is not any single result — it is that the same measurement machinery applies to weather, air quality, and power markets, and that it was committed to in writing before the data existed.
Collection began 2026-07-19. Hypotheses were pre-registered the same day, before any data
existed (PREREGISTRATION.md). The first scheduled analysis is
2026-07-22.
This README will not report a finding before its pre-registered gate. That restraint is the point of the exercise, so the empty results table below is a feature rather than an oversight.
| Domain | Collected so far | Coverage | First scheduled read |
|---|---|---|---|
| P-POP — NWS precipitation probability | 6,916 forecasts / 3,872 observations | 15 stations | 2026-08-09 (Brier/CORP) |
| P-GRID — day-ahead vs real-time power | 678 LMP / 48 load forecasts | CAISO (PJM pending key) | 2026-08-18 (H1/H2) |
| P-AQI — AirNow skill + revisions | 140 forecasts / 82 observations | 15 reporting areas | 2026-07-22 (revisions) |
Counts as of the snapshot in data/snapshots/. Collectors run on a
6-hourly / twice-daily / daily cadence respectively.
| Domain | Hypothesis | Verdict | n | Read on |
|---|---|---|---|---|
| — | — | pending first gate | — | — |
scoring/ proper scoring rules + reliability decomposition <- the reusable core
pop/ NWS PoP calibration analysis
grid/ day-ahead vs real-time spread analysis
aqi/ AirNow skill + revision analysis
data/ committed CSV snapshots + the export script
tests/ hand-computed fixture tests (no DB, no network)
Zero dependencies: no numpy, no pandas, no scipy. Every function is verified against hand-computed expected values, not against another library's output — the test comments show where each number comes from, so the tests are checkable by reading them.
- Proper scoring: Brier, log-loss
- Murphy decomposition: reliability / resolution / uncertainty, so a score is attributable rather than a single opaque number
- CORP reliability via pool-adjacent-violators, avoiding the arbitrary-bin-width degree of freedom that ordinary reliability diagrams smuggle in
- Bootstrap CIs, resampled at the level where dependence actually lives
No database, no API keys, no server. The snapshots are committed.
git clone https://github.com/AlexanderShields4/forecast-verification
cd forecast-verification
make check # lint + types + tests
make pop # writes reports/ from data/snapshots/data/snapshots/*.csv.gz are exact exports of the collection database, regenerated by
data/export_snapshots.sh. Sources: the NWS API, AirNow, and
CAISO/PJM via gridstatus — each polled on a fixed cadence and stored append-only, so a
revised value creates a new row rather than replacing the old one. That append-only property is
what makes the AirNow revision study (P-AQI question B) possible at all.
The ingestion pipeline itself — systemd timers, Postgres, backups, monitoring — lives in a separate private repository. This repo holds the science and the data needed to check it.
Given enough hypotheses and a fixed dataset, something always clears p < 0.05. The defense is to fix the questions, the minimum sample sizes, and the kill criteria before looking — and then to publish the null results with the same prominence as the positive ones.
So PREREGISTRATION.md was committed before the first row landed, and it carries explicit kill
criteria. One hypothesis (P-POP H1) is expected to confirm a known result — that one is the
control. If the pipeline cannot recover a forecast already known to be well-calibrated, the
pipeline is broken, and nothing else it produces should be trusted.
Amendments are appended, never edited in place.
MIT — see LICENSE.