Evaluate claims_all_entitled_benefits per benefit unit

[MaxGhenis]

Draft — held for a decision on the interim Council Tax Reduction take-up assumption (see "Decision to make").

What was wrong

claims_all_entitled_benefits ended its formula with add(benunit, period, [seven *_reported columns]).sum() < 1 — a sum over the whole array, collapsing an entity-level boolean into one simulation-wide scalar. On enhanced_frs_2024_25.h5 v1.56.14 at 2025 the flag was False for all 61,223 benefit units although 51,855 report under £1 in total across the seven benefits, so would_claim_council_tax_reduction (= flag OR reported CTR > 0) reduced to "reported CTR > 0" for every benefit unit, and simulated Council Tax Reduction reached FRS reporters only. Surfaced by the audit behind PolicyEngine/policyengine-uk-data#448.

This PR evaluates the test per benefit unit: a family reporting under £1 in total across the seven means-tested benefits is assumed to claim what it is entitled to; a family reporting £1 or more is assumed to have reported completely. Single-benefit-unit situations — the calculator path — are unchanged, since the two forms coincide there.

Why per benefit unit is the intended reading

• The variable is a benefit-unit attribute (entity = BenUnit, "whether this family would claim any benefit they are entitled to"), and tests set it per family.
• The simulation-wide False was deliberate in 2022: the formula and its comment arrived with the parametric-reform test infrastructure (c08d17c9), and tools/takeup.py (2022–2024, deleted in d380c0e8) derived take-up parameters from exactly that reporters-only baseline against a set_input(…, True) counterfactual.
• That reporters-only behaviour is now carried by each legacy benefit's own eligibility: housing_benefit_eligible, income_support_eligible, is_WTC_eligible and is_CTC_eligible each require the benefit's own reported amount to be positive (all four gates introduced in 13511284, 2022-03-23; those four benefits' random take-up branches were removed in 22bd7c8b, 2024-09-11). Measured on the licensed artefacts, the flag's value is immaterial to every one of those benefits' paid amounts (table below).
• would_claim_council_tax_reduction (#1769, June 2026) copied the housing benefit pattern, but the CTR chain has no reporter gate and CTR is open to new claims — so the collapsed flag silently restricted an active scheme to FRS reporters.

Measured consequences

Three behaviours were measured on the licensed artefacts (aggregates only; incumbent calibrated weights unchanged): the old formula, the per-benefit-unit test (this PR — verified field-for-field identical to running the new formula directly), and all-True (the historical "maximum claimants" counterfactual, as an upper bound).

Unchanged to the measured precision (£0.1m; identical unweighted recipient counts) in every measured cell (2024–2026 on v1.56.14; 2025 on v1.56.16): housing benefit, income support, working tax credit, child tax credit, income-based JSA, income-related ESA, universal credit, pension credit and child benefit. Of these, only housing benefit, income support, WTC and CTC consume the flag, and each is additionally gated by its own reported amount; the other five never read it (income-based JSA and income-related ESA have their own reporter-gated formulas, and UC, pension credit and child benefit use dataset take-up inputs).

Council Tax Reduction is the only paid amount that moves (2025, v1.56.14 → per benefit unit):

	old	per benefit unit	change
CTR, UK	£2.28bn	£3.95bn	+£1.67bn
CTR, England (pensioners + five local schemes)	£2.01bn	£3.09bn	+£1.08bn
CTR, Scotland	£0.25bn	£0.70bn	+£0.45bn
CTR, Wales	£0.01bn	£0.15bn	+£0.14bn
CTR recipients, UK	3.52m	4.75m	+1.24m
Net council tax, UK	£47.60bn	£45.94bn	−£1.67bn
Poverty, BHC	12.05%	11.89%	−0.16pp
Poverty, AHC	15.66%	15.49%	−0.17pp

The household-net-income change equals the CTR change to the stored precision in every v1.56.14 cell; on v1.56.16 at 2025 the two differ by one stored £0.1m unit (£1.6227bn vs £1.6228bn). v1.56.16 at 2025 is materially identical (UK CTR £2.29bn → £3.91bn). 2024: £2.21bn → £3.80bn; 2026: £2.33bn → £4.12bn.

Against official figures: Scotland's modelled CTR moves from £0.25bn (−36% against the ≈£0.389bn 2024-25 outturn, Scottish local government finance statistics 2024-25) to £0.70bn (+80%) — full take-up among entitled benefit units reporting none of the seven overshoots, as expected with no take-up rate applied. Wales moves from £0.01bn to £0.15bn against a 2024-25 scheme cost of £322.9m (Welsh CTR annual report); PolicyEngine/policyengine-uk-data#448 documents the imputed-savings gate and zero-liability records as the leading remaining Welsh gaps. No England or UK scheme-cost comparator was pinned.

Decision to make

The end state for CTR take-up is a dataset-provided would_claim_council_tax_reduction column anchored to official caseloads, the same way the data builds generate would_claim_uc, would_claim_pc and would_claim_child_benefit — policyengine-core prefers dataset inputs over formulas, so that follow-up needs no further model change. Until it exists, the interim choices are:

1. This PR: full take-up among benefit units reporting none of the seven benefits. Honest cost: Scotland overshoots its outturn by ~80% instead of undershooting by ~36%.
2. Alternative: keep CTR reporters-only explicitly until the data column lands — a one-line change to the same variable if preferred.

Follow-ups (out of scope)

• Data builds: generate a CTR take-up column (tracked against Scottish/Welsh council tax ~20% above devolved budget figures; WAS owned_land imputation unstable across releases policyengine-uk-data#448).
• The tax_credits intermediate is gated only by would_claim_WTC/would_claim_CTC, not by eligibility, so in 2024-and-earlier periods non-reporting families with children accrue a phantom intermediate (£0.53bn → £2.71bn at 2024) that dampens — never reverses — their new CTR awards through the means test. No paid tax-credit amount changes. From the labelled 2025 the scheme is inactive and the intermediate is zero.
• Calibration: the UK net council tax aggregate the data pipeline calibrates (council_tax_less_benefit) moves from £47.60bn to £45.94bn at 2025 on incumbent weights; recalibration happens in the data pipeline, not here.

Tests

• policyengine_uk/tests/test_claims_all_entitled_benefits.py: on the old formula the per-benefit-unit cases fail (e.g. [False, False] == [False, True]); the single-benefit-unit semantics cases pass unchanged before and after. A microsimulation-marked case asserts the flag equals the per-benefit-unit test record-for-record and takes both values in microdata.
• Full YAML policy suite and full pytest suite (including microsimulation-marked tests) pass locally, including the pensioner CTR case that exercises the formula without setting the flag.
• Independent verification (separate reviewer lanes): a reviewer lane recomputed 16 scenario/measure aggregates and matched the stored values, and running the committed formula reproduced the measured per-benefit-unit counterfactual in all 100 compared fields (v1.56.14, 2025).

References: PolicyEngine/policyengine-uk-data#448.

🤖 Generated with Claude Code