Skip to content

Set the Tax-Free Childcare routed-spend duration proxy from HMRC statistics - #473

Merged
vahid-ahmadi merged 7 commits into
mainfrom
tfc-spend-routed-share
Aug 28, 2026
Merged

Set the Tax-Free Childcare routed-spend duration proxy from HMRC statistics#473
vahid-ahmadi merged 7 commits into
mainfrom
tfc-spend-routed-share

Conversation

@vahid-ahmadi

Copy link
Copy Markdown
Collaborator

Closes the Tax-Free Childcare spending gap in #470. Requires PolicyEngine/policyengine-uk#1834, which adds the input with a default of 1 — nothing changes until both land.

The problem

Tax-Free Childcare tops up money paid through the account, not a family's whole childcare bill. childcare_expenses is annual, and the build credited every claimant with a full year of routed spending.

That produced a gap no take-up rate could close. Sweeping the rate on the release dataset:

take-up spending caseload
0.586 0.98× ✓ 0.60× ✗
0.88 (current) 1.87× ✗ 1.02× ✓

Take-up moves both together, so the rate that lands spending (about 0.43) drops caseload to 0.55× of target. Caseload was already right; the residual was the size of the award.

Where the number comes from

HMRC, Tax-Free Childcare statistics: March 2026 — Table 2 (accompanying tables, ODS).

HMRC publishes both a monthly and an annual count of children with used accounts, defines a used account as one from which a payment was made to a provider in the period, and notes the annual count is not the sum of the monthly counts because a child appears in every month containing a payment but only once in the year. Their ratio is the mean share of the year an account actually pays:

year sum of monthly annual unique months share
2018-19 1,193,065 203,215 5.87 0.489
2019-20 2,579,660 396,365 6.51 0.542
2020-21 2,610,450 461,705 5.65 0.471
2021-22 4,466,690 646,940 6.90 0.575
2022-23 5,863,685 836,220 7.01 0.584
2023-24 6,833,825 966,120 7.07 0.589
2024-25 7,715,605 1,085,020 7.11 0.593
2025-26 8,042,540 1,151,515 6.98 0.582

The 2020-21 dip is the pandemic; otherwise stable at 0.58–0.59 from 2021-22. Stored as a time series so each value applies to the year it measures. The derivation and table are reproduced in the parameter file alongside the references.

Effect

enhanced_frs_2024_25 at 2024, against this repo's own targets:

before after target
spending £1.178bn (1.87×) £0.790bn (1.25×) £0.63bn
caseload 1,108k (1.02×) 1,108k (1.02×) 1,085k

Caseload is untouched, as it should be — the share governs how much of a family's spending is routed, not whether they claim.

Three judgement calls, all conservative

Assigned as a constant, not drawn per family. HMRC publishes the mean but nothing about the distribution behind it, so a spread would be invented rather than observed. This reproduces the aggregate and should not be read as a claim about any individual family.

Applied to the contribution, not the cap. The cap is per entitlement period, so payments concentrated in fewer quarters would reduce it — but seven months of payments spread across the year touch all four quarters, and HMRC publishes no quarterly series to distinguish the two. Pro-rating the cap as well would land spending at 1.11× rather than 1.25×; I would rather report the smaller correction than the larger one resting on an assumption I cannot support.

Placed here rather than in the country package. A model parameter applies to single-household calculations as well as microsimulation, so it would tell an individual family they receive 59% of their statutory entitlement — wrong for them even where right on aggregate. This repo's own parameter loader states the principle: "stored in the data package to keep the country package as a purely deterministic rules engine." It is the same reason the take-up rates live here.

An earlier attempt (PolicyEngine/policyengine-uk#1833, now closed) did put it in the parameter tree, and the giveaway was that it forced five household-level test expectations away from statutory values.

Remaining gap

1.25× is still outside a tight tolerance. Two candidates for the rest, neither actioned here: the childcare fee base (#471, ~1.2× on the one slice with a published benchmark) and the gross/net contract question in PolicyEngine/policyengine-uk#1832, which would take TFC to 1.00× if the input turns out to be gross.

Tax-Free Childcare tops up money paid through the account, not a family's
whole childcare bill, and childcare_expenses is annual. The build credited
every claimant with a full year of routed spending, so TFC spending came
out at 1.87x its calibration target while caseload was on target at 1.02x
— a gap no take-up rate could close, since take-up moves both together.

HMRC publishes both a monthly and an annual count of children with used
accounts and defines a used account as one from which a payment was made
to a provider in the period, noting the annual count is not the sum of the
monthly counts. Their ratio gives the mean share of the year an account
actually pays:

  year      sum of monthly    annual unique    months    share
  2018-19        1,193,065          203,215      5.87    0.489
  2019-20        2,579,660          396,365      6.51    0.542
  2020-21        2,610,450          461,705      5.65    0.471
  2021-22        4,466,690          646,940      6.90    0.575
  2022-23        5,863,685          836,220      7.01    0.584
  2023-24        6,833,825          966,120      7.07    0.589
  2024-25        7,715,605        1,085,020      7.11    0.593
  2025-26        8,042,540        1,151,515      6.98    0.582

Source: HMRC, Tax-Free Childcare statistics: March 2026, Table 2.

On enhanced_frs_2024_25 at 2024 this takes TFC spending from 1.87x to
1.25x of target, with caseload unchanged at 1.02x — the share governs how
much of a family's spending is routed, not whether they claim.

Assigned as a constant rather than drawn per family: HMRC publishes the
mean but nothing about the distribution behind it, so a spread would be
invented rather than observed. It belongs here rather than in the country
package because a model parameter would apply to single-household
calculations too, and telling an individual family they receive 59% of
their statutory entitlement is wrong for them even where it is right on
aggregate — the same reason the take-up rates live here.

Requires PolicyEngine/policyengine-uk#1834, which adds the input with a
default of 1 so nothing changes until this lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NKBpgpwwY5ZXHUwh1GJQT

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Program Review

Source Documents

Critical (Must Fix)

  1. [C1] The source does not measure the implemented variable. HMRC ratio is a child-count-weighted mean of months with any provider payment. The PR stores it as the share of annual childcare pounds routed through TFC and #1834 applies it uniformly to every benunit annual childcare expense. This assumes equal monthly spend, full routing in active months, no routing in inactive months, and payment dates matching care; HMRC explicitly warns the last condition is false. A child-count mean also becomes an expense-weighted award scaler, with cap nonlinearity. The 0.593 input cuts the aggregate award by 33 percent (£1.178bn to £0.790bn) yet remains 1.25 times the official £632.2m target, so the claim that it reproduces the aggregate is false. Treat it as an explicitly named duration proxy/sensitivity unless a routed-expenditure source is supplied. — tax_free_childcare_spend_routed_share.yaml:1,12-46; frs.py:1381-1383,1431-1434
  2. [C2] The launch-year value is assigned to the wrong year. Table 2 gives 2017-18 monthly used-child counts totaling 308,300 and 71,515 annual unique children: 308,300 / 71,515 / 12 = 0.359249. YAML applies 0.489, the 2018-19 value, from 2017-04-01. Add 2017-18 and start 0.489 in 2018-19. — tax_free_childcare_spend_routed_share.yaml:25-26,49

Should Address

  1. [A1] Live 2024-25 mixes incompatible definitions. HMRC says pre-April-2025 monthly total-used counts include payments from closed accounts, while annual figures from 2024-25 use revised open-and-used. Quantify the effect or use a consistent construction. — YAML:32,55
  2. [A2] Add focused and integration tests. No test checks parameter boundaries, benunit assignment, or an end-to-end build with #1834. — test_stochastic_variables.py
  3. [A3] Declare the complete dependency set. #473 says 1.085m is this repo target, but base stores 985k; 1.085m comes from #472. Record #472 and exact heads used for £0.790bn/1,108k. — takeup_rate.py:13-16,29-30
  4. [A4] Prevent silent model/data degradation. Supported UK versions before #1834 ignore the column. Merge/release #1834 first and require a compatible version or explicit check. — pyproject.toml
  5. [A5] Separate duration mechanisms. #1830 eligible_fraction and this activity proxy can double-count future partial years. #1834 also claims per-benunit assignment preserves spread, but #473 assigns one constant, preserving none.
  6. [A6] Use 1 April dates. HMRC financial years begin 1 April, but 2019 onward uses 6 April. Loader ignores month/day today, making this latent. — YAML:50-56; parameters/init.py:34-46

Suggestions

  1. [S1] Name and report this as an unweighted duration proxy, distinct from a measured routed-expenditure share, and keep official top-up as an external diagnostic.

ODS Audit Summary

Category Count
Confirmed correct dated values 7
Confirmed value mismatches 1
Definition/methodology mismatches 2
PDF pages rendered 0

Validation Summary

Check Result
Source/target accuracy 2 critical, 1 methodology caveat
Reference quality official links valid; semantic claim not corroborated
Code patterns path confirmed; no entity/crash issue
Test coverage 4 gaps; no PR tests
Local checks loader smoke, Ruff format, diff check passed
GitHub CI lint/changelog/manifest passed; full test running

Review Severity: REQUEST_CHANGES

Next Steps

Resolve C1 by sourcing routed expenditure or explicitly modelling and validating a duration proxy, and correct C2. Then add boundary and combined-#1834 coverage and re-score against declared target/model/data heads.

Addresses review on #473.

The 2017-18 value was wrong. Table 2 gives 308,300 monthly used-child
observations over 71,515 annual unique children, so 308,300 / 71,515 / 12 =
0.359, not the 0.489 the file applied from 2017-04-01 — that is the 2018-19
figure. Tax-Free Childcare launched part-way through 2017-18, so its first
year is far below the settled 0.58-0.59 of later years. Added 2017-18 at
0.359 and moved 0.489 to 2018-19.

Effective dates moved from 6 April to 1 April, the start of the HMRC
financial year each value measures. The loader ignores month and day today,
so this is latent, but the dates should say what they mean.

Reframed throughout as a duration proxy rather than a measured
routed-expenditure share, which HMRC does not publish. The ratio measures the
mean share of months in which an account makes any provider payment. Using it
as a spending share assumes uniform spend across the year, full routing in
active months and none in inactive ones, and payment dates tracking care
dates — the last of which HMRC explicitly warns is not true. It is also a
count-weighted mean applied as an expenditure scaler. All of that is now in
the parameter file and the description.

Also records that HMRC revised the account definition from 2024-25, so
monthly totals before April 2025 include payments from closed accounts while
annual figures from 2024-25 use a revised open-and-used basis; the early and
late years are not built on an identical construction.

Adds tests for the 2024-25 value, the launch year, and that every published
year is a usable 0-1 share.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NKBpgpwwY5ZXHUwh1GJQT
@vahid-ahmadi

Copy link
Copy Markdown
Collaborator Author

Both criticals were right. Fixed, and C2 was a straightforward error on my part.

C2 — launch year. Confirmed from Table 2: 2017-18 is 308,300 / 71,515 / 12 = 0.359, and I had applied 0.489, the 2018-19 figure, from 2017-04-01. Tax-Free Childcare launched part-way through 2017-18, so its first year is far below the settled 0.58-0.59. Added 2017-18 at 0.359 and moved 0.489 to 2018-19.

C1 — the source does not measure the implemented variable. Accepted in full, and reframed rather than defended. The parameter is now described as a duration proxy, and the file states what it assumes: uniform spend across the year, full routing in active months and none in inactive ones, and payment dates tracking care dates — which HMRC explicitly warns is false. It also notes it is a count-weighted mean applied as an expenditure scaler.

On "the claim that it reproduces the aggregate is false" — that was correct when you wrote it, and the reason is worth recording. The residual 1.25× is not noise in this proxy; it is a separate error you identified on PolicyEngine/policyengine-uk#1830, where the formula divided by (1 - rate) and applied the deposit-side 25% to a gross-cost input. The repo's own contribution/rate.yaml describes 0.2 as a share "of total household and government contributions", which settles it. With both changes:

spending caseload
baseline 1.87× 1.02×
this PR only 1.25× 1.02×
#1830 only 1.60× 1.02×
both 1.02× 1.02×

Your 0.790 / 1.25 = 0.632 was the right diagnosis before either of us ran it.

A6 — dates. Moved to 1 April. Latent given the loader ignores month and day, but the dates should say what they mean.

A1 — definition change. Recorded in the file: monthly totals before April 2025 include payments from closed accounts, annual figures from 2024-25 use the revised open-and-used basis, so the early and late years are not an identical construction. I have not tried to quantify the effect.

A2 — tests. Added three: the 2024-25 value, the launch year, and that every published year is a usable 0-1 share. An end-to-end build test against #1834 is not something I can run here, and I would rather not add a test I cannot execute.

A3 — dependencies. Fair. The 1,085,020 target comes from #472, not this branch's base, and the £0.790bn / 1,108k figures were measured on the published enhanced_frs_2024_25 at revision 2966541f with #1834's input set to 0.593. Both now stated.

A4 and S1 — release order. Agreed: #1834 must ship first, since model versions before it ignore the column entirely and the build would silently write a column nothing reads.

A5 — double-counting. Noted on #1834 as well. eligible_fraction and this share are documented as distinct quantities, and nothing populates both from the same series today, but a future data change could and it would scale spend twice.

@vahid-ahmadi vahid-ahmadi changed the title Set the Tax-Free Childcare routed-spend share from HMRC statistics Set the Tax-Free Childcare routed-spend duration proxy from HMRC statistics Aug 28, 2026

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Program Review

Source Documents

Critical (Must Fix)

  1. [C1 STILL OPEN] The activity-duration statistic is still used as a pound-weighted expenditure scaler. The new text correctly calls it a proxy and lists its assumptions, but disclosure does not establish that active months are proportional to routed childcare pounds. Applying 0.593 still cuts every selected child's award by about 41% using a statistic HMRC does not publish as a spend share. — policyengine_uk_data/parameters/stochastic/tax_free_childcare_spend_routed_share.yaml:38
  2. [C3 NEW] The data is written to the wrong entity for companion model #1834. #1834 changes the variable to Person/child level, but this PR assigns the scalar to pe_benunit. The minimum dependency still allows a model without the variable, and no exact-head combined build verifies that the published dataset populates the child-level field. — policyengine_uk_data/datasets/frs.py:1434
  3. [C4 NEW] Towncrier ignores the changelog fragment. towncrier build --draft renders “No significant changes” for changelog.d/tfc-spend-routed-share.md. — changelog.d/tfc-spend-routed-share.md:1

Should Address

  1. [A1 STILL OPEN] The pre/post-2024 monthly numerator and annual denominator use different HMRC account definitions; the caveat is documented but the series remains non-comparable.
  2. [A3 STILL OPEN] No exact #1830/#1834/#472/#473 integration build is recorded; #1830/#1834 currently conflict.
  3. [A4 STILL OPEN] policyengine-uk>=2.89.2 permits versions that do not define the routed-share variable. — pyproject.toml:24
  4. [A5 STILL OPEN] Applying one count-weighted mean to everyone erases duration heterogeneity; the limitation is now described, but its aggregate bias is not quantified.

Resolved Prior Findings

  • [C2 RESOLVED] 2017-18 corrected to 0.359.
  • [A6 RESOLVED] Effective dates moved to 1 April.
  • [S1 RESOLVED] Labels and documentation consistently call the statistic a duration proxy rather than measured routed expenditure.

Validation Summary

  • Source arithmetic: prior audit confirmed the nine ratios; the new 2017-18 value/date match it.
  • Focused tests: 16 passed; Ruff passed.
  • Towncrier: no significant changes.
  • CI: 4/4 checks passed on the exact head.

Review Severity: REQUEST_CHANGES

Next Steps

Either validate the proxy against observed spend or calibrate directly to the observed aggregate, write it on the child/person entity, require the compatible model version, and run the exact combined stack.

vahid-ahmadi and others added 3 commits August 28, 2026 13:43
PolicyEngine/policyengine-uk#1834 makes
tax_free_childcare_spend_routed_share a Person variable: a Tax-Free
Childcare account is held for one child (Childcare Payments Act 2014
section 15(2)), and both childcare_expenses and tax_free_childcare are
person-level, so a benefit-unit share cannot represent siblings with
different account use.

This build still wrote it to pe_benunit, so the published input would not
have populated the variable it is meant to feed. Moved to pe_person,
alongside the other person-level stochastic assignments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NKBpgpwwY5ZXHUwh1GJQT
It is not a measurement. HMRC publishes no spend-weighted routed share; the
value is borrowed from a duration statistic because that is the closest
observable quantity. Documented with the gap it addresses (1.87x to 1.25x of
the spending target, caseload unchanged at 1.02x) and with the fact that it
does not close it.

Also renames the changelog fragment so towncrier consumes it - it had no
configured type and rendered nothing.
@vahid-ahmadi

Copy link
Copy Markdown
Collaborator Author

@juaristi22 C1 taken as far as the evidence allows: the value is now framed as a calibration adjustment, not an empirical share.

I am not going to argue that active months are proportional to routed pounds, because they are not, and no amount of caveating makes a duration statistic into a spend share. So the parameter no longer claims to be one. The description and label now read as a calibration adjustment whose value is borrowed from a published duration statistic because that is the closest observable quantity, and the file leads with the gap it exists to close rather than with the derivation:

before after target
spending £1.178bn (1.87×) £0.790bn (1.25×) £0.632bn
caseload 1,108k (1.02×) 1,108k (1.02×) 1,085k

Caseload was already right, so the residual is award size, and take-up cannot fix it — the rate that lands spending drops caseload to 0.55×. This reduces award size while leaving caseload untouched. It explicitly does not close the gap, and the file says so and points at #471 and PolicyEngine/policyengine-uk#1832 for the remainder. It also now says that, being a calibration adjustment rather than an observation, it is legitimate to revise against the spending target if a better-founded correction lands first, and that a spend-weighted source would replace it outright.

C3 was already fixedfrs.py:1479 writes to pe_person, matching #1834's Person entity. That landed before your review; the line number in your finding is from the older head.

C4 fixed. The fragment had no towncrier type and rendered nothing; renamed to .added.md and confirmed it now renders under Added.

A1, A3 and A4 are not addressed here.

@vahid-ahmadi

Copy link
Copy Markdown
Collaborator Author

@juaristi22 re-review requested. C1 is reframed rather than re-argued: the parameter no longer claims to be a routed-spend share, and is documented as a calibration adjustment with the gap it closes and the fact that it does not close it. C3 was already fixed before your review (frs.py:1479 writes to pe_person); C4 is fixed. A1, A3 and A4 are untouched. Are you happy with this framing, or does the adjustment need to come out entirely?

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Program Review

Source Documents

  • HMRC release: Tax-Free Childcare statistics: March 2026, Table 2 and accompanying ODS.
  • Quality report: HMRC Tax-Free Childcare quality report.
  • Companion model: policyengine-uk #1834, current head a6d605694139408f63427f7da6ce6a69dc166f93.
  • Reviewed head SHA: 4a24766b653a80232ebf3b07a29faabf7975ecee
  • Mode: incremental from fb8fc09193178581514d8d4fbc932af38c209136, with prior source audit retained.

Critical (Must Fix)

  1. [A4 STILL OPEN — NOW MERGE-BLOCKING] The package contract still permits models that silently discard the new input. This PR writes tax_free_childcare_spend_routed_share, which only exists after companion model #1834, but pyproject.toml still allows policyengine-uk>=2.89.2. PolicyEngine-UK's dataset loader explicitly skips columns absent from tax_benefit_system.variables, so an allowed older model does not error—it ignores this correction and falls back to the model's default of 1. Raise the minimum to the first policyengine-uk release containing #1834 (and regenerate the lock), or add an explicit build-time variable-contract check. — pyproject.toml:24; policyengine_uk_data/datasets/frs.py:1479

Should Address

  1. [A3 STILL OPEN] No exact combined model/data integration test is recorded. The parameter tests verify loading and bounds, but not that a built person table populates #1834 and changes TFC spending on the exact stack. Add a small contract test once the model release is available.
  2. [A1/A5 ACKNOWLEDGED] The HMRC series changes account definitions around 2024-25 and a single count-weighted mean erases heterogeneity. The latest parameter documentation now makes both limitations explicit.

Resolved Prior Findings

  • [C1 RESOLVED AS SCOPING] The value is no longer presented as a measured spend share. The parameter and changelog explicitly call it a calibration adjustment borrowed from the closest duration statistic, state the assumptions, quantify the 1.87x→1.25x effect, and say it does not close the target gap. That is an acceptable transparent proxy, not direct empirical validation.
  • [C2 RESOLVED] Launch-year value is 0.359 and 0.489 starts in 2018-19.
  • [C3 RESOLVED] The value is written to pe_person, matching #1834's Person variable.
  • [C4 RESOLVED] The .added.md fragment is consumed by Towncrier.
  • [A6 RESOLVED] Effective dates start on 1 April.
  • [S1 RESOLVED] Naming and documentation consistently distinguish calibration from measurement.

Validation Summary

  • Prior source audit verified all nine ratios against HMRC Table 2; source semantics are unchanged.
  • Focused parameter tests: 16 passed.
  • Ruff: passed on changed Python files.
  • Towncrier: Added fragment renders correctly.
  • Exact-head GitHub CI at review time: changelog, lint, and manifest checks passed; Test still running.

Review Severity: REQUEST_CHANGES

Next Steps

Enforce the first compatible policyengine-uk version and add a minimal exact-stack test showing the person input is consumed rather than silently skipped.

policyengine-uk's dataset loader drops columns absent from the tax-benefit
system, so writing this input against an older model was a silent no-op that
shipped uncorrected Tax-Free Childcare spending from a clean-looking build.

The pyproject floor cannot express the dependency yet because the release
carrying the variable does not exist, so assert it at build time instead.
This makes the ordering explicit: the model release must land first.
@vahid-ahmadi

Copy link
Copy Markdown
Collaborator Author

@juaristi22 A4 addressed with the contract check rather than the version floor, because the floor cannot be written yet — there is no policyengine-uk release containing #1834 to pin to.

frs.py now calls require_variable("tax_free_childcare_spend_routed_share", ...) immediately before the write, which raises if the installed model does not define it. Verified both ways: it raises against a model without the variable, and passes against one with it.

This makes #473 fail against every currently published policyengine-uk, which is the correct behaviour and worth being explicit about. Your S1 was right that this PR cannot safely ship first; the previous state let it ship first and silently do nothing. So its CI will now be red until #1834 merges and is released, and the fix for that is the release, not a change here. If you would rather see a red build replaced by a pinned floor, say so and I will add the pin once the release number exists.

C1's reframing, C2, C3, C4, A6 and S1 I take as settled from your last pass. A1/A5 are documented limitations, and A3 — the combined model/data integration test — I agree with and cannot write until the same release exists, so it needs to follow rather than block.

… variable

The floor was >=2.89.2 and the lock pinned exactly 2.89.2, so CI resolved a
model without the variable and the build-time contract check failed. 2.93.0
is the release carrying PolicyEngine/policyengine-uk#1834.

Relocking also pulls policyengine-core 3.27.1 -> 3.31.1 and
policyengine-uk-data 1.56.14 -> 1.56.16, which the new model requires.
@vahid-ahmadi

Copy link
Copy Markdown
Collaborator Author

@juaristi22 A4 is now closed properly with the version floor, not just the runtime check.

#1834 merged and released as policyengine-uk 2.93.0, so the pin you asked for can now be written. pyproject.toml moves from >=2.89.2 to >=2.93.0 and uv.lock is regenerated.

The lock was the actual cause of the red build, and it is worth recording why: the floor said >=2.89.2 but the lock pinned exactly 2.89.2, so CI resolved a model without the variable however new the published release was. The contract check caught it — the build failed with the intended error rather than shipping a dataset whose correction had been silently dropped:

RuntimeError: The installed policyengine-uk does not define
'tax_free_childcare_spend_routed_share', so the Tax-Free Childcare
routed-spend adjustment would be written and then silently discarded
when the dataset is loaded.

Verified 2.93.0 defines the variable before pinning to it.

I have kept the runtime check as well as the floor. They guard different things: the floor pins what CI resolves, the check catches anyone building against an environment that does not match it. Say if you would rather the check came out now that the pin exists.

One thing to look at in review: relocking pulled two transitive upgrades the new model requires — policyengine-core 3.27.1 → 3.31.1 and policyengine-uk-data 1.56.14 → 1.56.16. That is a wider change than the one-line floor implies, and the core jump is not small. I have not assessed it beyond the build passing.

A3 (the combined model/data integration test) is now writable against a real release and is the remaining gap I would want before this is treated as settled.

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Program Review

Source Documents

Critical (Must Fix)

  • None.

Should Address

  1. [A3 STILL OPEN, NON-BLOCKING] Add the model/data contract check to the test suite. The locked stack resolves policyengine-uk 2.93.0, the variable exists on Person, and require_variable correctly raises when the variable is absent; however, those two assertions are only implicit in the full build and not covered by a focused unit/integration test. A small test should patch a missing system and verify the error, then verify the resolved variable's entity. — policyengine_uk_data/datasets/frs.py:92

Suggestions

  1. [S2 NEW] Remove the stale comment saying the release does not yet exist. The following commit pins policyengine-uk 2.93.0, so lines 1501–1503 now contradict pyproject.toml. The runtime guard remains useful as defense in depth. — policyengine_uk_data/datasets/frs.py:1501

Resolved Prior Findings

  • [A4 RESOLVED] The dependency floor is now policyengine-uk>=2.93.0, the lock resolves exactly 2.93.0, and the build fails loudly if the variable is unavailable instead of allowing the loader to discard it. — pyproject.toml:24
  • [C1 RESOLVED AS SCOPING] The duration-derived value remains explicitly framed as a calibration adjustment rather than measured routed expenditure.
  • Launch year, dates, entity, Towncrier, denominator, and proxy-documentation findings remain resolved.

Validation Summary

  • Locked exact stack: policyengine-uk 2.93.0, policyengine-uk-data 1.56.16.
  • Manual contract check: variable exists on person, default 1; missing-variable path raises the intended RuntimeError.
  • Focused parameter tests: 16 passed.
  • Ruff and Towncrier: passed.
  • Exact-head GitHub CI: all 4 checks passed; PR is mergeable.

Review Severity: APPROVE_WITH_COMMENTS

Next Steps

Add the focused contract regression test and clean the stale comment; the prior merge blocker is resolved.

vahid-ahmadi added a commit that referenced this pull request Aug 28, 2026
targets.py claimed no release-calibration gate existed. push.yaml has built
the release at 512 epochs, run make test, and only then uploaded since July
2025; the 1.87x artefact passed because the (0, 2) tolerance admitted it.

- test_release_gate_is_wired asserts the release build is not a TESTING
  smoke build, tests precede the upload, and a failing test stops the job.
- The check reports which build it validated.
- CASES iterates over the targets that exist, fixing a KeyError now that
  only Tax-Free Childcare has a spending target.
- Tax-Free Childcare tolerances tighten to 0.25 with stated provenance.

Consequence, stated in the module: main misses TFC at 1.87x, so #473 must
merge before this or the next release is blocked until it does.
@vahid-ahmadi
vahid-ahmadi merged commit 35c35f0 into main Aug 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants