Adopt the UK local-geography and firms contracts and compile the local target surface (#708, #759)

[juaristi22]

Closes #708, closes #759.

Migrates the last two UK calibration-target contracts out of Chronicle per the
chronicle#166 ruling, and compiles the UK
local target surface from the wave-3 facts — the area-grain analog of #622, and the missing link
in the #495 local lane.

One PR for both issues by adjudication: #708's selector corrections are only proven once the local
facts compile, so the corrections and their compilation land together. The commit ladder keeps #708's
material first.

What lands

commit	scope
1	#708 — both contracts re-homed to build/uk/ with signed selector corrections, consumer re-pointed; plus the shared machinery the compile needs: the record_set_spec_id selector key, the declared count_x_mean operation, the local-area crosswalk, roster-driven area authoring, the generation CLI, country_spec loading and compile_uk_local_target_registry
2	#759 — the compiled local surface (references + membership) and the regenerated census
3	both compile-parity gates armed, every receipt row ruled, battery digests re-cut
4	the defect citation reworded to the form the live tree's incumbent-name guard allows
5	one population contract — uk_national_targets.json + uk_local_geography_targets.json merge into uk_population_targets.json (all 214 targets kept at their declared levels, per-target geography_levels; firms stay separate); local UC ids consolidate into dwp.uc.* as dwp.uc.households_by_area*, each recording renamed_from so the provenance chain to the retired Chronicle profile survives; the two accounting blocks partition rather than merge (registry_parity ×189, profile_parity ×25, partition-tested); the committed pair and receipt regenerate names-only — normalized payloads equal, citation counts identical
6	rebases onto the certified-cut main (#787's June-path retirement, then #793's release-cut certification): June-driver wiring retired with its substrate, gate ownership joined the closed-world scope partition, and the certifier — the landed runner our gates were scoped to — now compiles and supplies uk_ledger_compiled_local_registries beside the national key (the release-cut battery is 18 gates)
7	certification mirror re-pins: the release-cut part's frozen scope and digests re-derive for the 18-gate scope
8	review response — the fixture fix-and-signs uk-data#468 (name-joined UC, post-mapping attachment, recomputed splits; two new ruled drift classes), extractor code/length assertions, operative boundary-vintage accept-sets on the compile path, the geography-declaration closure test + runtime warning, and the restored national target_surface parameter forward

National regeneration: held, with the reason on the record

The plan's final step was to regenerate the national references at the refreshed feed. The dry run
came back worse (408 → 389 active) for two upstream causes, both diagnosed and both the
fail-closed machinery doing its job: Chronicle renamed the national UC-by-children source_concept
between feed vintages (breaking 10 selectors), and the new council-tax nation-total facts now also
match the 9 scotgov.council_tax_stock selectors (multi_fact). Committing that would have degraded
the national surface, so the committed national references keep their values and the refresh follows
chronicle#203 plus the selector updates in the lane that owns the national contract. The regenerated
evidence is preserved for that work.

The local surface

17,077 active references of 18,631 candidates over 650 constituencies and 361 local authorities
(25 contract targets × their declared levels), compiled from a sha-pinned Chronicle artifact
(108,112 rows, facts_sha256 4395a4e7…, Chronicle main 33ca98a).

Every absence is signed against measured feed coverage, never waved through:

family	outcome
ONS age ×8	resolved — 8,088 refs across all four nations
UC households + 4 child splits	GB-complete; 29 NI areas + 72 child-split rows signed absent (Stat-Xplore is GB-only)
SPI counts ×2	resolved (650 + 360 areas)
SPI amounts ×2	resolved via the declared count × mean operation — the publisher prints count/mean/median, never an amount
tenure ×4	resolved (per-nation census vintages)
equivalised income ×3	deferred — ONS publishes at MSOA grain only; the feed carries no LA rows
private rent ×1	deferred — every PIPR fact in the feed is dated 2026-06, after the 2025 target period
census_households	stays ladder-derived (#542); the compile records the ladder sha as provenance

A defect this compile exposed on our own side

The first compile pass showed the corrected age selectors still pinned source_name: "ons", which
excludes the NRS and NISRA publisher legs — the original Chronicle profile's central bug, carried one
step further. Dropping the pin resolves all 8 bands across England, Scotland, Wales and Northern
Ireland. This is exactly what compiling against real facts is for.

Parity gates

Two release-blocking preflight gates:

• uk_ledger_compile_parity_local_incumbent_2025 — value parity against a fixture extracted from
  policyengine-uk-data@12a1e028 by replaying both local loss builders' y-sides (23,545 rows, 38
  metrics), recording raw and calibrated values per row with each family's scaling factor so a
  difference can be attributed to the source statistic or to the incumbent's solve-time scaling.
• uk_target_surface_local_default_2025 — the structural check: the in-code metric_names(area_type)
  surface fanned over the crosswalk roster, reconciled against the compiled registry. No values are
  synthesized, because that surface has none.

All 23,834 receipt rows carry a ruled rationale; none keeps the classifier's generic text.

Two upstream findings, both filed

policyengine-uk-data#468 — the
incumbent's local UC targets are positionally misaligned, and the fixture fix-and-signs it. Name-only
spreadsheet rows are joined by row position to code files in a different order: 8 of 360 local
authorities land correctly, and 0 of 650 constituencies (the UC vector is PCON24-native yet paired
with 2010-ordered rows and then boundary-mapped on top). Birmingham calibrates to West Oxfordshire's
count, Hackney to the Isles of Scilly's. Verified by name-join, which reproduces the same Stat-Xplore
publication our facts carry at the incumbent's uniform national rescale — same data, permuted.

Per the reviewer's ruling (and the A&S row-30 precedent: fix-and-sign replaces
replicate-and-document), the fixture corrects the misalignment at extraction: UC counts join to
their areas by name, attach post-mapping on native PCON24/LAD codes, and the child splits recompute
from corrected totals with the incumbent's own country-proportion buckets keyed by true country. The
UC parity rows become interpretable for the first time, and split into two ruled classes: the 982
totals rows are the incumbent's uniform national rescale (fixture/ours 0.8925–0.9003), and the 2,525
child-split rows are an imputation-vs-published class — the incumbent imputes splits from GB country
shares while ours are the published per-area buckets, so the scatter (0.77–1.19) is the real
geographic variation in family size the imputation flattens. Three rows now match exactly.

The extraction refuses unresolved joins (650/650 and 360/360 areas must match by name), and the
crosswalk's declared boundary vintages are now operative on the compile path: a matched fact on a
non-equivalent boundary frame fails the compile by name, with equivalent publisher frames (ONS lists
devolved areas on its lad_2023 lookup over unchanged boundaries) declared as accept-sets.

chronicle#200 — two evidenced fact asks.
(1) Emit PIPR 2025: the package already preserves the whole sheet (49,266 source rows, 4,284/year for
2015–2025) and only selected_rows narrows fact emission to 2026-06; calendar-2025 carries all 12
months with 316/316 E&W local authorities, so widening it closes 314 of the 361 rent deferrals with no
new acquisition. (2) Port NI Universal Credit from the DfC May-2025 supplementary tables the incumbent
already binds, closing the 29 NI totals.

Incumbent left-behind audit

The incumbent binds 38 metric columns, not 25. The 13 outside our contract are signed exclusions
with ruled rationales: 4 devolved private-rent columns whose y-values are hardcoded constants in
devolved_housing.py with no source or year, and 9 council-tax columns the contract does not declare
(the wave-3 facts exist; the microcosm-side family work is recorded as follow-up scope).

Also confirmed by the audit: the NOMIS ASHE workbook feeds no loss matrix, the SPI
pension/total-income columns never bind, and there is no age target above 79.

Adjudications recorded

Selector corrections land here as fix-and-sign with a profile_parity accounting block; one shared
feed refresh with the national movement signed in this PR; SPI amounts bind through a declared
operation rather than deferring; the committed reference pair keeps the literal national row shape
(12 MB references + 11 MB membership, measured and accepted, rather than a compact encoding).

Deferred, deliberately

The national-surface refresh is held, with its cause filed. Regenerating the national references
against the current feed was attempted and came back worse (408 → 389 active): Chronicle's
April–December UC packages renamed the family's source_concept strings, and the new SLGFS
council-tax yield rows reuse the ctaxbase band measure ids, so 19 targets defer — correctly, under
the fail-closed compile. Both causes are upstream and filed as
chronicle#203; the refresh follows once it
lands and a fresh feed is staged. The committed national references are untouched by this PR. (The
local surface is unaffected — the per-area packages kept their vocabulary, which is how its 17,077
references compile on this same feed.)

The support oracle (#495 increment 4's measurement half) needs licensed microdata and is not in
this PR. chronicle#172 retires the
Chronicle-side profile surface after this merges — that sequencing is unchanged, and it is why the
consumer adoption lands first.

🤖 Generated with Claude Code

[vahid-ahmadi]

Automated review pass (Claude Code, high effort, diff only — no execution). Code hunks reviewed; the ~1.05M lines of generated JSON were not read row by row.

Four findings, and the first is a merge-block.

1. tools/build_uk_ledger_compile_parity_signed_differences.py:~109 — policyengine-uk-data#468 is replayed here, inside the fixture the parity gate trusts

uc = get_constituency_uc_targets().values attaches UC counts to rows by position (same for the uc_children[col].values loop below), and line ~139 then pushes the whole frame through mapping_matrix @ raw.values — a 2010→PCON24 transform — while codes is read independently from constituencies_2024.csv. That is the #468 defect exactly: UC data already on 2024 boundaries is boundary-mapped a second time, and no key ties a UC value to its constituency.

What makes this worse than the upstream instance is where it sits. This file generates the reference fixture the compile-parity gate compares against, so the misalignment is baked in as the expected answer: the gate then passes on permuted values, and a green parity result is evidence of nothing on those columns. A future correct compilation would read as a parity failure against this fixture.

Worth noting the fixture cannot be regenerated correctly until #468 itself is fixed, since it sources from the misaligned getters — so this is coupled to that issue rather than independently fixable.

2. Same file, rows_for (~273) — nothing asserts the orderings agree

Each area row is emitted by for i, code in enumerate(codes) indexing raw[col].values[i] / cal[col].values[i]. The income, age and UC blocks are all built from .values of separately-fetched frames carrying no code column, and nothing asserts len(codes) == len(raw) or that the two orderings correspond. A partial or reordered re-extract of any upstream getter silently produces a permutation instead of failing.

The contrast is in the same file: the ONS-income, tenure, rent and council-tax blocks all use explicit merge(left_on="code", right_on="la_code"). As with uk-data's loss builders, the correct pattern is already present a few lines away.

3. uk_runtime/ledger_targets.py:~494 and measure_simulation.py:~304 — a missing field reads as "national"

The new national filter is set(target.get("geography_levels") or ()) <= {"country", "region"}, and the empty set is a subset of everything — so a target with a missing or empty geography_levels is classified as national. Now that uk_population_targets.json also holds local rows, any local target that omits the field leaks into the national contract rather than being excluded. Requiring the field, and refusing a target that does not declare it, would make an omission loud instead of defaulting it into the wrong surface.

4. uk_runtime/battery_bindings.py:~659 — a dropped forward turns working entries into hard failures

_evaluate_target_surface no longer forwards **dict(parameters) to uk_target_surface_gate, and now raises ValueError for any non-empty parameters other than expected="local_default_surface". Any existing national target_surface gate entry that declares parameters — reviewed exclusions, for instance — goes from working to hard-failing on the first run after this lands.

Also worth fixing while you are here

_local_crosswalk_rosters reads expected_vintage and only interpolates it into an error message; no reference's declared boundary vintage is ever compared against it. So the PCON24-vs-2010 declaration is decorative on the compile path — the one check that would have caught finding 1 structurally is present in name only.

---

Finding 1 blocks: a parity gate whose reference fixture encodes the permutation cannot certify the thing it is being cited for. 3 and 4 are ordinary bugs that will show up on first run. 2 is the durable one — adding the merge and a totality assertion is what stops this class recurring, and chronicle#202 shows the shape it should take (every row carrying its geography_id, expected_row_header and guard_cells so a reordered extract fails the guard instead of permuting).

[juaristi22]

Thanks — all four findings and the closing note are dispositioned in adbb9125.

1 (merge-block) — accepted, resolved beyond the ask. The reviewer's ruling on this thread
overturned the fixture's replicate-and-document posture (the A&S row-30 precedent: fix-and-sign
replaces it): the fixture now corrects uk-data#468 at extraction rather than reproducing it. UC
counts join to their areas by name, attach after the boundary mapping on native PCON24/LAD codes
(never through the 2010→2024 matrix — the data is 2024-native), and the child splits recompute from
corrected totals with the incumbent's own country-proportion buckets keyed by true country. Your
"cannot be regenerated correctly until #468 is fixed" no longer holds: the correction is the name
join the upstream getters lack, applied fixture-side and signed. The UC rows become interpretable and
split into two ruled classes — 982 totals rows at the incumbent's uniform national rescale
(fixture/ours 0.8925–0.9003) and 2,525 child-split rows as imputation-vs-published (their splits are
GB-share imputations; ours are the published per-area buckets; the 0.77–1.19 scatter is real
geographic family-size variation). Three rows now match exactly.

2 — accepted. Code/length assertions on every positionally-consumed source: code-keyed families
(income, age) assert their code column against the roster in order; UC totals must resolve
650/650 and 360/360 by name or the extraction refuses; the boundary mapping's shape is asserted
against both rosters.

3 — accepted with a ruling. A missing geography_levels defaulting to national is doctrine (the
empty set is national by construction), so the fix is the omission being impossible-then-loud rather
than a refusal: a closure test now requires every committed target to declare the field, and both
filter sites warn if a hand-built contract ever omits it.

4 — accepted. The parameter forward is restored on the national path; declared parameters reach
uk_target_surface_gate unchanged and unknown ones still fail closed inside the gate. (At head the
national entry declares no parameters, so nothing was live-broken, but the regression was real.)

Closing note — accepted, and it caught something on its first run. The crosswalk's boundary
vintages are now operative: a matched fact on a non-equivalent frame fails the compile by name. The
first live run refused SPI facts at NI districts stamped lad_2023 — which turned out to be a
publisher-vocabulary equivalence, not a defect: ONS lists devolved areas on its 2023 LAD lookup over
boundaries unchanged since ca_2019/lgd_2014. The crosswalk now declares those as explicit
accept-sets; a vintage outside the set still refuses.

Dispositions in adbb9125; the lint follow-up (strict=True on the new name-join zips) is d1fc092d.

🤖 Generated with Claude Code

[vahid-ahmadi]

Re-review of adbb9125 + d1fc092d (Claude Code, high effort, diff only — no execution). I looked at the fixes rather than the PR, since they land in the component that decides whether the compile-parity gate means anything.

The corrective posture is right, and it is a better outcome than what I asked for. Correcting #468 at extraction — name join, attach after the boundary mapping on native PCON24/LAD codes, child splits recomputed from corrected totals — makes the fixture an independent statement of truth rather than a mirror of the upstream defect, and it decouples this PR from the uk-data fix. Splitting the UC rows into the two ruled classes (982 totals at the uniform national rescale, 2,525 child-split rows as imputation-vs-published) is what makes the remaining scatter interpretable instead of suspicious.

Four gaps in the new code. None of them undo that; all four are the same shapes as the original round, one layer in.

1. tools/extract_uk_local_registry_fixture.py:~215 — the corrected child splits do not sum back to the corrected total

The splits are built as round(corrected[code] * shares[j]) per bucket, each rounded independently, so the four uc_hh_* columns need not sum to uc_households — every area can be off by up to ±2 households. Either a closure check asserting splits == total fails, or no such check exists and the fixture ships internally inconsistent, which is the worse case for a reference artifact.

Largest-remainder allocation gives buckets that sum exactly to the corrected total. Worth adding the closure assertion alongside it, since "the parts sum to the whole" is the kind of property a fixture should prove about itself rather than inherit.

2. tools/extract_uk_local_registry_fixture.py:~100 — _assert_code_aligned degrades to a length check

The guard runs only if "code" in frame.columns. For any positionally-consumed source frame that lacks a code column — the income getters, if they carry names only — it falls back to a bare length comparison, which passes for any permutation of the right size.

That is the #468 failure mode surviving inside the assertion written to prevent it, and it is the same shape as finding 2 from the first round: the correct pattern is present, but the path that skips it is silent. A missing code column should refuse rather than fall through — if a source cannot be code-checked, that is a fact about the source worth failing on, not a case to wave past.

3. packages/microcosm-build/src/microcosm/build/uk_runtime/ledger_targets.py:~232 — the vintage gate passes unstamped facts

if vintage and vintage not in accepted lets a fact with an absent or empty geography.vintage through silently. Combined with the two neighbouring escapes — if wanted is None: continue for an unknown code prefix, and if not expected: return when the level is not in the rosters — an unstamped or oddly-prefixed fact binds across vintages unchecked.

Making the vintages operative is exactly right, and the NI lad_2023 accept-sets are a good outcome from the first live run: a real publisher-vocabulary equivalence found by a check that had been decorative. But the gate exists to prove the frame, and a fact that declines to say which frame it is on is the case it most needs to catch. An unstamped fact should refuse.

4. tools/extract_uk_local_registry_fixture.py:~65 — the name join is checked in one direction only

Every roster area must resolve, which covers the missing case. Publisher rows that match no roster area are dropped without error, so a superset or a renamed-area extract passes silently. Asserting len(by_name) == len(roster) — 650/650 and 360/360 in both directions — closes it.

Related: duplicate-key detection runs on the normalized key, so two genuinely distinct areas that collide after split(" / ")[0].lower() raise rather than resolving. That is the safe direction to fail in, but the message will describe a duplicate where the real condition is a normalization collision, which is worth distinguishing for whoever hits it.

---

1 is the one I would fix before merge: a reference fixture whose parts do not sum to its own totals is a weak reference regardless of how well aligned it is. 2 is the durable one — for the same reason as last round, the guard that silently skips is more dangerous than no guard, because it reads as coverage.

[juaristi22]

All four dispositioned in 8b8d4e41 — accepted, and your framing ("the guard that silently skips reads as coverage") is the right lens on every one of them.

1 — accepted, fixed before merge as asked. Largest-remainder allocation with a per-area closure assertion: the four buckets now sum exactly to the corrected total, and the fixture proves the property about itself rather than inheriting it. Re-extraction moved exactly 225 split rows by at most 1 household each; nothing else changed, and the receipt counts are stable (23,834).

2 — accepted. A positionally-consumed source without a code column now refuses outright — "cannot be proven code-aligned" is a fact about the source worth failing on, as you put it. Both income getters carry codes, so the refusal is pure fence on the real sources.

3 — accepted, all three escapes. An unstamped fact, an unknown code prefix, and a level with no declared vintage each refuse by name. The unstamped case got your wording in the error text, since it is the case the gate most needs to catch.

4 — accepted, both halves. The join asserts totality in both directions (unmatched publisher rows fail with their names listed), and a normalization collision is now distinguished from a true duplicate in the error, so whoever hits it sees the real condition.

The commit also carries a rebase onto 4b133a8e with the spec pin re-derived, so the regeneration ran once on the final base.

🤖 Generated with Claude Code

requesting review again @vahid-ahmadi

[vahid-ahmadi]

Third pass over 8b8d4e41 and the rebase (Claude Code, high effort). This one checked the committed artifacts rather than reasoning from the diff, since the round-2 fixes were mostly claims about data.

The round-2 fixes hold, and I verified rather than assumed:

• Closure. Parsed local_registry_parity_fixture_2025.json out of the diff: 650/650 constituencies have exactly four uc_hh_* rows summing to uc_households, zero mismatch, all totals integral. The largest-remainder loop is arithmetically sound — the deficit total - sum(floors) is provably in [0, 4], so no negative slice and no under-allocation, and zero totals allocate all-zero cleanly.
• Name join. _corrected_uc_by_code is genuinely bijective: roster→publisher via the missing list, publisher→roster via len(by_name) != len(roster) with orphan names listed, duplicates raising before either, and zip(..., strict=True) closing the length escape. The bilingual Welsh handling (split(" / ")[0]) is applied identically to both sides, so it cannot desynchronize them.
• Positional guard. _assert_code_aligned orders length → code column presence → exact order, and no path reaches a bare length check. Both surfaces' positionally-consumed frames (incomes, age_targets) go through it.
• Vintage gate. All three named refusals are reachable. I went looking for a fourth — a fact whose geography block is absent entirely — and it is unreachable in practice: _local_candidate_fact_pool only returns facts bucketed by _local_fact_geography_key, which requires a Mapping geography with string level and id, and _assert_local_reference_in_crosswalk proves the selector carries both first. Non-string vintages are stringified and then fail the accept-set rather than passing.
• Rebase and regeneration. Self-consistent. counts_by_kind (16782 + 6760 + 292) equals difference_count 23834; fixture_count 23545 equals the fixture's own row_count. The uk_national_targets.json → uk_population_targets.json rename is complete — no code, resource manifest or generator references the old name — and no pre-rebase digest survives.

Two residuals and one thing worth confirming.

1. tools/extract_uk_local_registry_fixture.py (constituency_surface, the if sum(floors) != total block) — the closure assertion is tautological

Two problems in the same few lines. It re-sums the list the preceding lines just constructed to equal total, so it cannot fail on its own construction; and it asserts against total = round(corrected[code]) while the value written to the fixture is the unrounded corrected[code] float.

So the property you added it for — the buckets sum to the stored uc_households — is still unasserted. It is correct today only because all 650 totals happen to be integral (I checked), and would silently drift by up to 0.5 households the first time the publisher ships a non-integral count. Asserting the buckets against the value actually written, rather than against a locally-rounded copy, is the version that proves the claim.

Flagging this one specifically because it is the fix for round 2's finding 1 arriving as a check that cannot fail — the exact class the round was about. Easy to do accidentally: the assertion reads as a closure check and is positioned as one.

2. packages/microcosm-build/src/microcosm/build/target_reference_authoring.py (author_area_target_references) — a level the roster lacks drops out silently

if area_ids is None: continue skips any contract target declaring a geography_level the crosswalk roster does not carry: no candidates, no references, no error — and _target_status([]) now returns "not_applicable", so the membership report shows nothing amiss either. Inert today, since the crosswalk carries both levels, but it is the one place a whole level could drop out of the surface without a refusal, and the status mapping means the report would agree that nothing was wrong.

Worth confirming: 360 against 361

The fixture LA roster is 360 areas with 10 NI districts, while the crosswalk declares 361 and the signed deferral rationales say "361 local authorities" and "the 11 Northern Ireland local authorities". These are different artifacts and the gap lands in the signed ledger_only bucket, so it may well be deliberate — but an off-by-one against a rationale that states a specific count is worth an explicit check that the extra NI district is the intended exclusion rather than a dropped row that happens to land in a signed bucket.

---

Nothing here blocks in the way round 1's finding 1 did. 1 is worth fixing because it is the assertion the last round asked for and it currently proves nothing; 2 and the 360/361 question are both "confirm and move on" unless something surprising turns up.

[juaristi22]

All three dispositioned in 74a6e6fc.

1 — accepted, and thank you for the precision of the diagnosis. The assertion now compares the allocated buckets against the value the fixture actually stores (corrected[code], unrounded), not the locally-rounded copy — so it holds today because every publisher total is integral, and refuses loudly the day one is not, instead of drifting half a household. Re-extraction under the tightened assert is byte-identical. You are right that it was round 2's fix arriving as a check that could not fail; the same class, one layer in, again.

2 — accepted. A declared level the roster lacks now refuses by target and level instead of skipping into not_applicable. Worth recording why this is safe under the merged population contract: the generator already pre-filters targets to roster-covered levels (_filter_contract_by_geography_levels), so national country/region targets never reach the authoring core — the continue was unreachable-except-by-accident, exactly your "inert today" read. Regenerating the real surface with the refusal in place is byte-identical at 17,077 active.

360/361 — confirmed, by name. The incumbent's local_authorities_2021.csv carries N09000001–N09000010 and omits N09000011 (Newry, Mourne and Down) entirely — a genuine incumbent roster gap, not a dropped row on our side. The ledger_only rationale now states the district by code and name rather than describing the gap generically.

🤖 Generated with Claude Code