Skip to content

K=1 rowwise pilot machinery: clone the spine, assign the ladder, measure per-area support (#761) - #815

Open
juaristi22 wants to merge 11 commits into
mainfrom
uk-rowwise-pilot-761
Open

K=1 rowwise pilot machinery: clone the spine, assign the ladder, measure per-area support (#761)#815
juaristi22 wants to merge 11 commits into
mainfrom
uk-rowwise-pilot-761

Conversation

@juaristi22

@juaristi22 juaristi22 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Closes #761 (#495 increment 5, workstream G of #665).

The K=1 rowwise pilot machinery: clone the microcosm-built spine, assign geography through the
ratified OA ladder, and measure per-area support — no solve, no binding, no publication. The epic's
K table was sized against the retired 535,080-row certified compact; the spine that replaced it is
52,846 households, so K re-adjudicates from measured support. This PR lands the measurement
machinery; the licensed runs follow on this branch (two rounds per María's ruling: spine-i now as
shakedown + control, spine-j for the K ruling once #750's licensed phase lands).

What lands

commit scope
1 the local-areas line renames to uk-local-* and its uk/local Logbook scope is ratified — migration + DECLARED_SCOPES mirror + README move together (María's naming ruling: singular uk/local)
2 the household_id mod 10^8 lineage rule restates for the spine: the pool-only modulus is refused on inputs carrying explicit lineage columns, and the plan/manifest gain an additive explicit lineage block (distinct sources, per-channel splits, flag counts, pre-clone grain)
3 the ladder clone path refuses pre-assigned geography columns — stale *_oa columns cannot ride through beside the ladder's own outputs
4 fail-closed --input-sha256 / --ladder-sha256 pins (calibration-seam convention: refuse before any bytes parse or side effect), with an informational cross-check against the committed local_area_crosswalk.json ladder pin
5 expected_uk_ladder_area_support — the exact analytic expectation (constituency household-count share within region × input region mix × K; OA population shares for LA), sharing the sampler's own stage-weight helpers so it cannot drift from the draw
6 per-area diagnostics lift off the solve matrix: uk_area_support_summary (one groupby, numerically identical), matrix-free uk_ladder_area_support_summary, uk_region_mix; ladder manifest + dry-run gain area_support / region_mix blocks and the real build writes area_support_summary.csv
7 --candidate-clone-counts plans rows/bytes/support at every candidate K in one dry-run (dry-run-only surface, refused on a real build); every candidate K runs the identical extracted clone/assign/gate pipeline
8 the driver's docstring names the spine as the sanctioned input — never the certified compact — plus the changelog fragment

What does not change

No solve and no binding: build_uk_rowwise_candidate.py untouched beyond the two-line pipeline
rename, local_doctrine.py byte-identical, fences and adjudication registers untouched. The
Logbook rung stays f100 — rungs are sample-fraction tokens (#624 grammar) and the pilot consumes
100% of the spine at any K. Dry-runs still append no Logbook row. Weight fences (kind carriage,
mass-log currency, exact conservation at 1e-9, the single clone mass record, the write-path
re-gate) already did what #761 items 3–4 require and are untouched.

Still to land on this branch

  • Round A (spine-i, 0cce0399…) LANDED: the multi-K dry-run plan and the real K=1 build
    both ran on the licensed machine — ladder gate passed 650/650 constituencies and 361/361 LAs,
    zero missing-geography rows, mass conserved exactly (abs_delta 0.0), IMPORTANCE kind carried
    with one appended mass record, 16,288 distinct sources via the explicit lineage columns, 17 s /
    1.51 GB peak RSS. Constituency min-ESS by K: 15.3 (K=1) → 28.8 → 86.4 → 245 (K=10). The genesis
    uk/local Logbook row is archived and the chain validates. Receipts:
    experiments/761-uk-rowwise-pilot-receipts.md (+ per-K stats JSON); recommendation there is
    K=4 for First calibrated rowwise UK local candidate (#495 increment 6) #762, adjudication pending Round B.
  • Round B (spine-j): re-run when UK WAS stage: split private pension wealth out of corporate_wealth (uk-data#452 M2) #750's licensed phase re-cuts the acceptance receipt; the K
    ruling cites spine-j with spine-i as control.
  • The written K recommendation with its measurement, for María to adjudicate; the Adjudicate the UK spine grain at the ingest/calibration seam #688 comment
    (clone-count-as-declaration + measured pre/post-clone grain) posts from the receipts.

Note for the reviewer: the hosted-store migration (20260829000000_logbook_uk_local_scope.sql)
needs manual application; until then the licensed run's live insert warns and continues — local
spooling and the git export path are unaffected.

Verification

Ruff clean; ci_test_groups.py --verify ok; the full uk group ran green pre-rebase (the two
test_uk_local_area_crosswalk failures were the moved #795 branch, resolved by rebasing onto the
merged main); post-rebase the delta surface (16 files: logbook, rowwise, ladder, local-rowwise,
crosswalk, census, spec-pin suites) is green. PR CI is the authoritative full run.

🤖 Generated with Claude Code

juaristi22 and others added 10 commits August 29, 2026 02:22
The rowwise drivers' pipeline names move from uk-locals-* to uk-local-*
so the derived chain scope is uk/local (Maria's naming ruling on #761),
and the scope is ratified for the local-areas line's first archived run:
the scope_declared migration, the DECLARED_SCOPES mirror, and the README
vocabulary move together, per the closed-world convention. No archived
rows existed under the old names (the scope was never ratified), so the
rename is history-free.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The household_id-mod-10^8 rule was written for the certified pool's
tier-offset id scheme. The microcosm spine's ids stack flag-driven
offsets (sernum + 1{spi}.10^d + 1{cgt_clone}.10^(d+1) +
1{band_donor}.10^(d+2)) and carry explicit lineage columns, so
apply_uk_source_lineage_modulus now refuses any input carrying them --
the columns are the source of truth, and a derived modulus layer would
mint a false second lineage. The dry-run plan and build manifest gain an
additive explicit lineage block (distinct source households, per-channel
splits, flag counts, computed at pre-clone grain), null for pool inputs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
assign_uk_geography_ladder silently overwrites oa/lsoa/msoa codes and
would leave stale *_oa columns riding through beside its own outputs.
The ladder clone path and its dry-run plan now refuse a household input
that already carries any ladder geography column or legacy *_oa name
(region, the required enum input, excepted) -- two disagreeing
constituency columns on one artifact is exactly the laundering the
fences exist to prevent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pilot's contract is a sha-pinned spine and a sha-pinned ladder. Both
pins follow the calibration seam's convention (measured-vs-pinned
refusal before any bytes are parsed or output side effect happens), and
each artifact block records pin_verified. The ladder block additionally
cross-checks the measured sha against the committed
local_area_crosswalk.json ladder_artifact_sha256 pin -- informational,
never a refusal, so synthetic test ladders keep working while a real run
records whether it matched the ratified artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
expected_uk_ladder_area_support computes exact expected rows per
constituency (household-count share within region x input region mix x
n_clones) and per local authority (OA population shares within
constituency), sharing the sampler's own stage-weight helpers so the
expectation cannot drift from the draw. The ladder dry-run plan gains an
expected_support block beside realized_support, zero-support areas
included. The stage-two zero-mass refusal now names the constituency
instead of surfacing as an rng.choice probability error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
uk_area_support_summary is the frame-agnostic core (one groupby instead
of the per-area scan, numerically identical); the matrix-coupled
rowwise_area_support_summary becomes a thin wrapper with its signature
untouched. The new uk_ladder_area_support_summary computes the same
per-area ESS / distinct-source table for a plain ladder-assigned frame
on the full ladder roster, and uk_region_mix reports the input's region
composition as shares. The ladder route's manifest and dry-run plan gain
area_support (min/median rows, ESS, distinct sources, bottom-15 by rows
and by ESS, with the source_basis recorded) and region_mix blocks, and
the real build writes the full table as area_support_summary.csv.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--candidate-clone-counts (dry-run only, refused on a real build before
any side effect) plans every candidate K independently at the build
seed: per-K rows and byte estimates on both routes, realized and
analytic expected support, and on the ladder route the per-area support
stats. The fenced ladder plan body is extracted into
_ladder_planned_assignment so the primary plan and every candidate K run
the identical clone/assign/gate pipeline. The crosswalk route's
candidate blocks omit area_support (no ladder roster) and say so in
their basis string.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One attempt, disposition iterating: uk-local-rowwise ladder route, f100,
seed 42, spine-i (0cce0399) through the ratified ladder (9c6d56b9) --
gate passed 650/650, zero missing-geography rows, mass conserved
exactly. Row digest a4d2c0b2. The chain opens at this row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on spine-i: the multi-K plan (K in {1,2,4,10}) and the real K=1
build. 650/650 constituencies at every K; constituency min-ESS climbs
15.3 -> 28.8 -> 86.4 -> 245; the analytic expectation tracks the
realized draw on both grains; the K=1 build conserves mass exactly at
1.51 GB peak RSS. Recommendation: K=4 for the first calibrated
candidate; adjudication pending Round B (spine-j) per Maria's ruling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_committed_archives_are_scoped_by_country pins the committed archive
inventory; archiving the genesis uk/local row legitimately extends it.
Caught by CI's engine-free lanes -- the local runs predated the archive
commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22
juaristi22 marked this pull request as ready for review August 29, 2026 02:27
@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Automated review pass (Claude Code, high effort, diff only — no execution; test hunks skipped). Three findings.

1. uk_runtime/local_rowwise.py:1046 and tools/build_uk_rowwise_dataset.py:1573 — the support stats count zero-weight synthetics as support

_area_support_stats reports min_rows, median_rows and bottom_by_rows from assigned_households, which counts every assigned row — including the 10,000 zero-weight SPI synthetic households and their half-mass CGT twins. uk_area_support_summary computes nonzero_households immediately alongside, and the stats block drops it.

So the headline per-area support figures overstate real coverage, and the two numbers in the same summary disagree about the same area: one whose only rows are zero-weight synthetics reports a positive min_rows while its min_ess is 0. That is the specific failure mode this measurement exists to catch — an area that looks supported and is not — and the bottom_by_rows ranking, which is presumably how a thin area gets noticed, is the part most affected.

Since nonzero_households is already computed, reporting the stats over it (or reporting both, explicitly labelled) seems the small fix. Worth deciding what "support" means for the synthetic channel generally: those rows exist to shape the taxpayer population, not to carry area mass, so counting them as area support looks wrong in every consumer of this figure rather than just this one.

2. tools/build_uk_rowwise_dataset.py:422 — a pinned build can proceed unverified

--ladder-sha256 is only refused when --crosswalk is explicitly given. On the crosswalk-generated route (args.crosswalk is None, args.ladder is None) the flag is silently ignored and _verify_artifact_pin is never reached.

So passing a pin on that route does not verify anything and does not complain — the build proceeds as though unpinned. A caller who supplied a digest believes they got verification they did not get, which is worse than not offering the flag on that route at all. Refusing the flag as unsupported there, or wiring the verification through, both close it; silently ignoring a pin is the one option that misleads.

3. tools/build_uk_rowwise_dataset.py:1216,1288 — discarded per-area frames in both dry-run paths

_ladder_area_support_diagnostics is called in both dry-run paths with its second return value bound to _area_support_long and discarded, after the concat of both long frames has been built — full per-area frames for ~650 constituencies and ~360 local authorities, once per candidate K. The work is done and thrown away. Either consume it or stop building it.


1 is the one I would fix before merge: it is a measurement that reports coverage an area does not have, in the machinery whose purpose is deciding whether per-area support is adequate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

K=1 rowwise pilot on the microcosm-built spine: clone, assign geography, measure per-area support (#495 increment 5)

2 participants