Set the Tax-Free Childcare routed-spend duration proxy from HMRC statistics - #473
Conversation
Tax-Free Childcare tops up money paid through the account, not a family's whole childcare bill, and childcare_expenses is annual. The build credited every claimant with a full year of routed spending, so TFC spending came out at 1.87x its calibration target while caseload was on target at 1.02x — a gap no take-up rate could close, since take-up moves both together. HMRC publishes both a monthly and an annual count of children with used accounts and defines a used account as one from which a payment was made to a provider in the period, noting the annual count is not the sum of the monthly counts. Their ratio gives the mean share of the year an account actually pays: year sum of monthly annual unique months share 2018-19 1,193,065 203,215 5.87 0.489 2019-20 2,579,660 396,365 6.51 0.542 2020-21 2,610,450 461,705 5.65 0.471 2021-22 4,466,690 646,940 6.90 0.575 2022-23 5,863,685 836,220 7.01 0.584 2023-24 6,833,825 966,120 7.07 0.589 2024-25 7,715,605 1,085,020 7.11 0.593 2025-26 8,042,540 1,151,515 6.98 0.582 Source: HMRC, Tax-Free Childcare statistics: March 2026, Table 2. On enhanced_frs_2024_25 at 2024 this takes TFC spending from 1.87x to 1.25x of target, with caseload unchanged at 1.02x — the share governs how much of a family's spending is routed, not whether they claim. Assigned as a constant rather than drawn per family: HMRC publishes the mean but nothing about the distribution behind it, so a spread would be invented rather than observed. It belongs here rather than in the country package because a model parameter would apply to single-household calculations too, and telling an individual family they receive 59% of their statutory entitlement is wrong for them even where it is right on aggregate — the same reason the take-up rates live here. Requires PolicyEngine/policyengine-uk#1834, which adds the input with a default of 1 so nothing changes until this lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NKBpgpwwY5ZXHUwh1GJQT
juaristi22
left a comment
There was a problem hiding this comment.
Program Review
Source Documents
- Official tables: HMRC March 2026 accompanying ODS, Table 2
- Definitions: HMRC TFC quality report
- PDF: none published; HMRC supplies HTML plus ODS
- Year: 2017-18 through 2025-26; live FRS base year 2024
- Scope: PR changes plus cross-repo paths through UK #1830/#1834 and sibling data #472
- Reviewed head SHA: d960d2b
- Mode: full
Critical (Must Fix)
- [C1] The source does not measure the implemented variable. HMRC ratio is a child-count-weighted mean of months with any provider payment. The PR stores it as the share of annual childcare pounds routed through TFC and #1834 applies it uniformly to every benunit annual childcare expense. This assumes equal monthly spend, full routing in active months, no routing in inactive months, and payment dates matching care; HMRC explicitly warns the last condition is false. A child-count mean also becomes an expense-weighted award scaler, with cap nonlinearity. The 0.593 input cuts the aggregate award by 33 percent (£1.178bn to £0.790bn) yet remains 1.25 times the official £632.2m target, so the claim that it reproduces the aggregate is false. Treat it as an explicitly named duration proxy/sensitivity unless a routed-expenditure source is supplied. — tax_free_childcare_spend_routed_share.yaml:1,12-46; frs.py:1381-1383,1431-1434
- [C2] The launch-year value is assigned to the wrong year. Table 2 gives 2017-18 monthly used-child counts totaling 308,300 and 71,515 annual unique children: 308,300 / 71,515 / 12 = 0.359249. YAML applies 0.489, the 2018-19 value, from 2017-04-01. Add 2017-18 and start 0.489 in 2018-19. — tax_free_childcare_spend_routed_share.yaml:25-26,49
Should Address
- [A1] Live 2024-25 mixes incompatible definitions. HMRC says pre-April-2025 monthly total-used counts include payments from closed accounts, while annual figures from 2024-25 use revised open-and-used. Quantify the effect or use a consistent construction. — YAML:32,55
- [A2] Add focused and integration tests. No test checks parameter boundaries, benunit assignment, or an end-to-end build with #1834. — test_stochastic_variables.py
- [A3] Declare the complete dependency set. #473 says 1.085m is this repo target, but base stores 985k; 1.085m comes from #472. Record #472 and exact heads used for £0.790bn/1,108k. — takeup_rate.py:13-16,29-30
- [A4] Prevent silent model/data degradation. Supported UK versions before #1834 ignore the column. Merge/release #1834 first and require a compatible version or explicit check. — pyproject.toml
- [A5] Separate duration mechanisms. #1830 eligible_fraction and this activity proxy can double-count future partial years. #1834 also claims per-benunit assignment preserves spread, but #473 assigns one constant, preserving none.
- [A6] Use 1 April dates. HMRC financial years begin 1 April, but 2019 onward uses 6 April. Loader ignores month/day today, making this latent. — YAML:50-56; parameters/init.py:34-46
Suggestions
- [S1] Name and report this as an unweighted duration proxy, distinct from a measured routed-expenditure share, and keep official top-up as an external diagnostic.
ODS Audit Summary
| Category | Count |
|---|---|
| Confirmed correct dated values | 7 |
| Confirmed value mismatches | 1 |
| Definition/methodology mismatches | 2 |
| PDF pages rendered | 0 |
Validation Summary
| Check | Result |
|---|---|
| Source/target accuracy | 2 critical, 1 methodology caveat |
| Reference quality | official links valid; semantic claim not corroborated |
| Code patterns | path confirmed; no entity/crash issue |
| Test coverage | 4 gaps; no PR tests |
| Local checks | loader smoke, Ruff format, diff check passed |
| GitHub CI | lint/changelog/manifest passed; full test running |
Review Severity: REQUEST_CHANGES
Next Steps
Resolve C1 by sourcing routed expenditure or explicitly modelling and validating a duration proxy, and correct C2. Then add boundary and combined-#1834 coverage and re-score against declared target/model/data heads.
Addresses review on #473. The 2017-18 value was wrong. Table 2 gives 308,300 monthly used-child observations over 71,515 annual unique children, so 308,300 / 71,515 / 12 = 0.359, not the 0.489 the file applied from 2017-04-01 — that is the 2018-19 figure. Tax-Free Childcare launched part-way through 2017-18, so its first year is far below the settled 0.58-0.59 of later years. Added 2017-18 at 0.359 and moved 0.489 to 2018-19. Effective dates moved from 6 April to 1 April, the start of the HMRC financial year each value measures. The loader ignores month and day today, so this is latent, but the dates should say what they mean. Reframed throughout as a duration proxy rather than a measured routed-expenditure share, which HMRC does not publish. The ratio measures the mean share of months in which an account makes any provider payment. Using it as a spending share assumes uniform spend across the year, full routing in active months and none in inactive ones, and payment dates tracking care dates — the last of which HMRC explicitly warns is not true. It is also a count-weighted mean applied as an expenditure scaler. All of that is now in the parameter file and the description. Also records that HMRC revised the account definition from 2024-25, so monthly totals before April 2025 include payments from closed accounts while annual figures from 2024-25 use a revised open-and-used basis; the early and late years are not built on an identical construction. Adds tests for the 2024-25 value, the launch year, and that every published year is a usable 0-1 share. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NKBpgpwwY5ZXHUwh1GJQT
|
Both criticals were right. Fixed, and C2 was a straightforward error on my part. C2 — launch year. Confirmed from Table 2: 2017-18 is 308,300 / 71,515 / 12 = 0.359, and I had applied 0.489, the 2018-19 figure, from 2017-04-01. Tax-Free Childcare launched part-way through 2017-18, so its first year is far below the settled 0.58-0.59. Added 2017-18 at 0.359 and moved 0.489 to 2018-19. C1 — the source does not measure the implemented variable. Accepted in full, and reframed rather than defended. The parameter is now described as a duration proxy, and the file states what it assumes: uniform spend across the year, full routing in active months and none in inactive ones, and payment dates tracking care dates — which HMRC explicitly warns is false. It also notes it is a count-weighted mean applied as an expenditure scaler. On "the claim that it reproduces the aggregate is false" — that was correct when you wrote it, and the reason is worth recording. The residual 1.25× is not noise in this proxy; it is a separate error you identified on PolicyEngine/policyengine-uk#1830, where the formula divided by
Your A6 — dates. Moved to 1 April. Latent given the loader ignores month and day, but the dates should say what they mean. A1 — definition change. Recorded in the file: monthly totals before April 2025 include payments from closed accounts, annual figures from 2024-25 use the revised open-and-used basis, so the early and late years are not an identical construction. I have not tried to quantify the effect. A2 — tests. Added three: the 2024-25 value, the launch year, and that every published year is a usable 0-1 share. An end-to-end build test against #1834 is not something I can run here, and I would rather not add a test I cannot execute. A3 — dependencies. Fair. The 1,085,020 target comes from #472, not this branch's base, and the £0.790bn / 1,108k figures were measured on the published A4 and S1 — release order. Agreed: #1834 must ship first, since model versions before it ignore the column entirely and the build would silently write a column nothing reads. A5 — double-counting. Noted on #1834 as well. |
juaristi22
left a comment
There was a problem hiding this comment.
Program Review
Source Documents
- HMRC release: Tax-Free Childcare statistics: March 2026
- Quality report: HMRC Tax-Free Childcare quality report
- Reviewed head SHA:
0d51936965abb4d0ca2d2cebec045d7401171bc8 - Mode: incremental from
d960d2b25a72fb34e3bc0c8a1f10e4cbae08bad0
Critical (Must Fix)
- [C1 STILL OPEN] The activity-duration statistic is still used as a pound-weighted expenditure scaler. The new text correctly calls it a proxy and lists its assumptions, but disclosure does not establish that active months are proportional to routed childcare pounds. Applying 0.593 still cuts every selected child's award by about 41% using a statistic HMRC does not publish as a spend share. —
policyengine_uk_data/parameters/stochastic/tax_free_childcare_spend_routed_share.yaml:38 - [C3 NEW] The data is written to the wrong entity for companion model #1834. #1834 changes the variable to Person/child level, but this PR assigns the scalar to
pe_benunit. The minimum dependency still allows a model without the variable, and no exact-head combined build verifies that the published dataset populates the child-level field. —policyengine_uk_data/datasets/frs.py:1434 - [C4 NEW] Towncrier ignores the changelog fragment.
towncrier build --draftrenders “No significant changes” forchangelog.d/tfc-spend-routed-share.md. —changelog.d/tfc-spend-routed-share.md:1
Should Address
- [A1 STILL OPEN] The pre/post-2024 monthly numerator and annual denominator use different HMRC account definitions; the caveat is documented but the series remains non-comparable.
- [A3 STILL OPEN] No exact #1830/#1834/#472/#473 integration build is recorded; #1830/#1834 currently conflict.
- [A4 STILL OPEN]
policyengine-uk>=2.89.2permits versions that do not define the routed-share variable. —pyproject.toml:24 - [A5 STILL OPEN] Applying one count-weighted mean to everyone erases duration heterogeneity; the limitation is now described, but its aggregate bias is not quantified.
Resolved Prior Findings
- [C2 RESOLVED] 2017-18 corrected to 0.359.
- [A6 RESOLVED] Effective dates moved to 1 April.
- [S1 RESOLVED] Labels and documentation consistently call the statistic a duration proxy rather than measured routed expenditure.
Validation Summary
- Source arithmetic: prior audit confirmed the nine ratios; the new 2017-18 value/date match it.
- Focused tests: 16 passed; Ruff passed.
- Towncrier: no significant changes.
- CI: 4/4 checks passed on the exact head.
Review Severity: REQUEST_CHANGES
Next Steps
Either validate the proxy against observed spend or calibrate directly to the observed aggregate, write it on the child/person entity, require the compatible model version, and run the exact combined stack.
PolicyEngine/policyengine-uk#1834 makes tax_free_childcare_spend_routed_share a Person variable: a Tax-Free Childcare account is held for one child (Childcare Payments Act 2014 section 15(2)), and both childcare_expenses and tax_free_childcare are person-level, so a benefit-unit share cannot represent siblings with different account use. This build still wrote it to pe_benunit, so the published input would not have populated the variable it is meant to feed. Moved to pe_person, alongside the other person-level stochastic assignments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NKBpgpwwY5ZXHUwh1GJQT
It is not a measurement. HMRC publishes no spend-weighted routed share; the value is borrowed from a duration statistic because that is the closest observable quantity. Documented with the gap it addresses (1.87x to 1.25x of the spending target, caseload unchanged at 1.02x) and with the fact that it does not close it. Also renames the changelog fragment so towncrier consumes it - it had no configured type and rendered nothing.
|
@juaristi22 C1 taken as far as the evidence allows: the value is now framed as a calibration adjustment, not an empirical share. I am not going to argue that active months are proportional to routed pounds, because they are not, and no amount of caveating makes a duration statistic into a spend share. So the parameter no longer claims to be one. The description and label now read as a calibration adjustment whose value is borrowed from a published duration statistic because that is the closest observable quantity, and the file leads with the gap it exists to close rather than with the derivation:
Caseload was already right, so the residual is award size, and take-up cannot fix it — the rate that lands spending drops caseload to 0.55×. This reduces award size while leaving caseload untouched. It explicitly does not close the gap, and the file says so and points at #471 and PolicyEngine/policyengine-uk#1832 for the remainder. It also now says that, being a calibration adjustment rather than an observation, it is legitimate to revise against the spending target if a better-founded correction lands first, and that a spend-weighted source would replace it outright. C3 was already fixed — C4 fixed. The fragment had no towncrier type and rendered nothing; renamed to A1, A3 and A4 are not addressed here. |
|
@juaristi22 re-review requested. C1 is reframed rather than re-argued: the parameter no longer claims to be a routed-spend share, and is documented as a calibration adjustment with the gap it closes and the fact that it does not close it. C3 was already fixed before your review ( |
juaristi22
left a comment
There was a problem hiding this comment.
Program Review
Source Documents
- HMRC release: Tax-Free Childcare statistics: March 2026, Table 2 and accompanying ODS.
- Quality report: HMRC Tax-Free Childcare quality report.
- Companion model: policyengine-uk #1834, current head
a6d605694139408f63427f7da6ce6a69dc166f93. - Reviewed head SHA:
4a24766b653a80232ebf3b07a29faabf7975ecee - Mode: incremental from
fb8fc09193178581514d8d4fbc932af38c209136, with prior source audit retained.
Critical (Must Fix)
- [A4 STILL OPEN — NOW MERGE-BLOCKING] The package contract still permits models that silently discard the new input. This PR writes
tax_free_childcare_spend_routed_share, which only exists after companion model #1834, butpyproject.tomlstill allowspolicyengine-uk>=2.89.2. PolicyEngine-UK's dataset loader explicitly skips columns absent fromtax_benefit_system.variables, so an allowed older model does not error—it ignores this correction and falls back to the model's default of 1. Raise the minimum to the first policyengine-uk release containing #1834 (and regenerate the lock), or add an explicit build-time variable-contract check. —pyproject.toml:24;policyengine_uk_data/datasets/frs.py:1479
Should Address
- [A3 STILL OPEN] No exact combined model/data integration test is recorded. The parameter tests verify loading and bounds, but not that a built person table populates #1834 and changes TFC spending on the exact stack. Add a small contract test once the model release is available.
- [A1/A5 ACKNOWLEDGED] The HMRC series changes account definitions around 2024-25 and a single count-weighted mean erases heterogeneity. The latest parameter documentation now makes both limitations explicit.
Resolved Prior Findings
- [C1 RESOLVED AS SCOPING] The value is no longer presented as a measured spend share. The parameter and changelog explicitly call it a calibration adjustment borrowed from the closest duration statistic, state the assumptions, quantify the 1.87x→1.25x effect, and say it does not close the target gap. That is an acceptable transparent proxy, not direct empirical validation.
- [C2 RESOLVED] Launch-year value is 0.359 and 0.489 starts in 2018-19.
- [C3 RESOLVED] The value is written to
pe_person, matching #1834's Person variable. - [C4 RESOLVED] The
.added.mdfragment is consumed by Towncrier. - [A6 RESOLVED] Effective dates start on 1 April.
- [S1 RESOLVED] Naming and documentation consistently distinguish calibration from measurement.
Validation Summary
- Prior source audit verified all nine ratios against HMRC Table 2; source semantics are unchanged.
- Focused parameter tests:
16 passed. - Ruff: passed on changed Python files.
- Towncrier: Added fragment renders correctly.
- Exact-head GitHub CI at review time: changelog, lint, and manifest checks passed; Test still running.
Review Severity: REQUEST_CHANGES
Next Steps
Enforce the first compatible policyengine-uk version and add a minimal exact-stack test showing the person input is consumed rather than silently skipped.
policyengine-uk's dataset loader drops columns absent from the tax-benefit system, so writing this input against an older model was a silent no-op that shipped uncorrected Tax-Free Childcare spending from a clean-looking build. The pyproject floor cannot express the dependency yet because the release carrying the variable does not exist, so assert it at build time instead. This makes the ordering explicit: the model release must land first.
|
@juaristi22 A4 addressed with the contract check rather than the version floor, because the floor cannot be written yet — there is no policyengine-uk release containing #1834 to pin to.
This makes #473 fail against every currently published policyengine-uk, which is the correct behaviour and worth being explicit about. Your S1 was right that this PR cannot safely ship first; the previous state let it ship first and silently do nothing. So its CI will now be red until #1834 merges and is released, and the fix for that is the release, not a change here. If you would rather see a red build replaced by a pinned floor, say so and I will add the pin once the release number exists. C1's reframing, C2, C3, C4, A6 and S1 I take as settled from your last pass. A1/A5 are documented limitations, and A3 — the combined model/data integration test — I agree with and cannot write until the same release exists, so it needs to follow rather than block. |
… variable The floor was >=2.89.2 and the lock pinned exactly 2.89.2, so CI resolved a model without the variable and the build-time contract check failed. 2.93.0 is the release carrying PolicyEngine/policyengine-uk#1834. Relocking also pulls policyengine-core 3.27.1 -> 3.31.1 and policyengine-uk-data 1.56.14 -> 1.56.16, which the new model requires.
|
@juaristi22 A4 is now closed properly with the version floor, not just the runtime check. #1834 merged and released as policyengine-uk 2.93.0, so the pin you asked for can now be written. The lock was the actual cause of the red build, and it is worth recording why: the floor said Verified 2.93.0 defines the variable before pinning to it. I have kept the runtime check as well as the floor. They guard different things: the floor pins what CI resolves, the check catches anyone building against an environment that does not match it. Say if you would rather the check came out now that the pin exists. One thing to look at in review: relocking pulled two transitive upgrades the new model requires — A3 (the combined model/data integration test) is now writable against a real release and is the remaining gap I would want before this is treated as settled. |
juaristi22
left a comment
There was a problem hiding this comment.
Program Review
Source Documents
- HMRC release: Tax-Free Childcare statistics: March 2026, Table 2 and accompanying ODS.
- Quality report: HMRC Tax-Free Childcare quality report.
- Companion model release: policyengine-uk
2.93.0, containing #1834. - Reviewed head SHA:
50e66b3c3ed8be3e94191e5f3af3af22615740f3 - Mode: incremental from
4a24766b653a80232ebf3b07a29faabf7975ecee.
Critical (Must Fix)
- None.
Should Address
- [A3 STILL OPEN, NON-BLOCKING] Add the model/data contract check to the test suite. The locked stack resolves policyengine-uk 2.93.0, the variable exists on
Person, andrequire_variablecorrectly raises when the variable is absent; however, those two assertions are only implicit in the full build and not covered by a focused unit/integration test. A small test should patch a missing system and verify the error, then verify the resolved variable's entity. —policyengine_uk_data/datasets/frs.py:92
Suggestions
- [S2 NEW] Remove the stale comment saying the release does not yet exist. The following commit pins policyengine-uk 2.93.0, so lines 1501–1503 now contradict
pyproject.toml. The runtime guard remains useful as defense in depth. —policyengine_uk_data/datasets/frs.py:1501
Resolved Prior Findings
- [A4 RESOLVED] The dependency floor is now
policyengine-uk>=2.93.0, the lock resolves exactly 2.93.0, and the build fails loudly if the variable is unavailable instead of allowing the loader to discard it. —pyproject.toml:24 - [C1 RESOLVED AS SCOPING] The duration-derived value remains explicitly framed as a calibration adjustment rather than measured routed expenditure.
- Launch year, dates, entity, Towncrier, denominator, and proxy-documentation findings remain resolved.
Validation Summary
- Locked exact stack: policyengine-uk
2.93.0, policyengine-uk-data1.56.16. - Manual contract check: variable exists on
person, default 1; missing-variable path raises the intendedRuntimeError. - Focused parameter tests:
16 passed. - Ruff and Towncrier: passed.
- Exact-head GitHub CI: all 4 checks passed; PR is mergeable.
Review Severity: APPROVE_WITH_COMMENTS
Next Steps
Add the focused contract regression test and clean the stale comment; the prior merge blocker is resolved.
targets.py claimed no release-calibration gate existed. push.yaml has built the release at 512 epochs, run make test, and only then uploaded since July 2025; the 1.87x artefact passed because the (0, 2) tolerance admitted it. - test_release_gate_is_wired asserts the release build is not a TESTING smoke build, tests precede the upload, and a failing test stops the job. - The check reports which build it validated. - CASES iterates over the targets that exist, fixing a KeyError now that only Tax-Free Childcare has a spending target. - Tax-Free Childcare tolerances tighten to 0.25 with stated provenance. Consequence, stated in the module: main misses TFC at 1.87x, so #473 must merge before this or the next release is blocked until it does.
Closes the Tax-Free Childcare spending gap in #470. Requires PolicyEngine/policyengine-uk#1834, which adds the input with a default of 1 — nothing changes until both land.
The problem
Tax-Free Childcare tops up money paid through the account, not a family's whole childcare bill.
childcare_expensesis annual, and the build credited every claimant with a full year of routed spending.That produced a gap no take-up rate could close. Sweeping the rate on the release dataset:
Take-up moves both together, so the rate that lands spending (about 0.43) drops caseload to 0.55× of target. Caseload was already right; the residual was the size of the award.
Where the number comes from
HMRC, Tax-Free Childcare statistics: March 2026 — Table 2 (accompanying tables, ODS).
HMRC publishes both a monthly and an annual count of children with used accounts, defines a used account as one from which a payment was made to a provider in the period, and notes the annual count is not the sum of the monthly counts because a child appears in every month containing a payment but only once in the year. Their ratio is the mean share of the year an account actually pays:
The 2020-21 dip is the pandemic; otherwise stable at 0.58–0.59 from 2021-22. Stored as a time series so each value applies to the year it measures. The derivation and table are reproduced in the parameter file alongside the references.
Effect
enhanced_frs_2024_25at 2024, against this repo's own targets:Caseload is untouched, as it should be — the share governs how much of a family's spending is routed, not whether they claim.
Three judgement calls, all conservative
Assigned as a constant, not drawn per family. HMRC publishes the mean but nothing about the distribution behind it, so a spread would be invented rather than observed. This reproduces the aggregate and should not be read as a claim about any individual family.
Applied to the contribution, not the cap. The cap is per entitlement period, so payments concentrated in fewer quarters would reduce it — but seven months of payments spread across the year touch all four quarters, and HMRC publishes no quarterly series to distinguish the two. Pro-rating the cap as well would land spending at 1.11× rather than 1.25×; I would rather report the smaller correction than the larger one resting on an assumption I cannot support.
Placed here rather than in the country package. A model parameter applies to single-household calculations as well as microsimulation, so it would tell an individual family they receive 59% of their statutory entitlement — wrong for them even where right on aggregate. This repo's own parameter loader states the principle: "stored in the data package to keep the country package as a purely deterministic rules engine." It is the same reason the take-up rates live here.
An earlier attempt (PolicyEngine/policyengine-uk#1833, now closed) did put it in the parameter tree, and the giveaway was that it forced five household-level test expectations away from statutory values.
Remaining gap
1.25× is still outside a tight tolerance. Two candidates for the rest, neither actioned here: the childcare fee base (#471, ~1.2× on the one slice with a published benchmark) and the gross/net contract question in PolicyEngine/policyengine-uk#1832, which would take TFC to 1.00× if the input turns out to be gross.