Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions PROGRESS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Behavior input boundary progress

## State

- Branch: `feat/be-behavior-input-contract`
- Starting base: `3c3b4f6442f4a5adc47274734d71a6ca10103b43`
- Local `origin/main` at start: `3c3b4f6442f4a5adc47274734d71a6ca10103b43`
- Live `origin/main` was fetched on 2026-08-31 and remains exactly
`3c3b4f6442f4a5adc47274734d71a6ca10103b43`.
- Live comparison: this worktree's common repository is shallow and therefore
counts 1,161 commits at `origin/main`. An independent non-shallow clone has
the same `origin/main` SHA with 1,168 commits, matching GitHub's live history
count. The apparent seven-commit difference was shallow history, not an
upstream advance; the exact comparison base remains
`3c3b4f6442f4a5adc47274734d71a6ca10103b43`.
- The local `main` ref is unrelated divergent work and is not a safe upstream
substitute.
- Scope: data-only behavior input configuration and ID-keyed resolution; no
simulation or country-computation integration.
- Frozen-review status: locally ready. The code, tests, documentation, type
check, Ruff, diff, and issue-bound Towncrier checks are green. Draft
publication and independent frozen-head review remain.

## Done

- Verified the worktree was clean and the requested branch, base, local
`origin/main`, and merge-base matched exactly.
- Read all repository instruction files, the relevant engineering skills, and
the complete defensive correctness audit.
- Completed read-only review of `YearData`, Pydantic, pandas, tests,
documentation, changelog, and package conventions.
- Added the focused behavior-input tests first. The isolated test run is red at
collection because the not-yet-implemented public models are absent, as
expected. The normal `uv run` environment could not be created offline, so
the red run used the available Python environment with repository conftests
disabled; later green validation is recorded below.
- Resumed from the TDD commit and inspected the dirty implementation, salvage
ref `refs/codex-salvage/feat-be-behavior-input-contract-20260830-212607-7535`,
complete architecture audit, and all repository instruction files. The dirty
implementation is byte-identical to the salvage snapshot.
- Confirmed the requested boundary has no dependency on `Simulation.run()`, a
legal-variable registry, positional entity mapping, or source runtimes.
- Added the frozen, extra-forbidding `BehaviorInputBinding` and `BehaviorInputs`
models, exported only those configuration models from `policyengine.core`,
and kept the pure resolver internal.
- The resolver requires non-null unique stable entity IDs, returns copied
ID-indexed series, preserves nullable values, and does not mutate source
tables or remap entities.
- Focused diagnostic validation passes all 17 contract tests under the available
Python 3.14 environment. Direct Ruff format/check and `git diff --check` pass
for the implementation slice. A syncing `uv run` remains blocked because the
sandbox cannot write the configured user cache; later validation therefore
uses `uv run --no-sync` with an existing development environment.
- Hardened the source-aliasing test against MicroDataFrame scalar-setter
differences across pandas/microdf versions and corrected the Ruff import
grouping. The 17 focused tests and direct Ruff checks remain green.
- Documented cross-system ownership, legal/behavioral vocabulary, the
stable-ID resolution contract, and the explicit execution non-goals in the
model architecture guide.
- Stronger validation through `uv run --no-sync` and the canonical clone's
existing development environment is green: 17 focused behavior tests; 36
Belgium/labor-supply tests with 4 expected source-stack skips; and 73
dataset, model, and extra-variable regression tests.
- Whole-repository Ruff format-check and lint pass (`197 files already
formatted`; no lint findings), and both working-tree and branch diffs pass
`git diff --check`.
- The architecture Markdown parses successfully through Quarto's Pandoc. The
full `make docs` render is environment-blocked because Quarto attempts to
open its Sass database in a non-writable user cache, not because of a
documentation diagnostic.
- GitNexus reported that this worktree was not indexed; its index attempt was
blocked by the non-writable global registry. The generated untracked index
artifacts were removed, and impact review used direct source/history instead.
- Final architecture review confirmed the source/API scope is clean and added
the audit's exact legal-operability, non-inference, and explicit non-goal
guardrails to the documentation.
- Focused mypy validation passes for `src/policyengine/core/behavior.py`, and a
runtime API probe reconfirms JSON round-trip, private resolver scope, and no
top-level `pe.be` export.
- Towncrier comparison now finds `changelog.d/510.added.md` and passes against
the freshly fetched `origin/main`.
- Independent final reviews found no blocking code or architecture defect and
assessed implementation risk as low. The previously missing issue-numbered
Towncrier fragment is now resolved by issue #510.
- Created PolicyEngine/policyengine.py issue #510, added
`changelog.d/510.added.md`, and re-fetched the live upstream base without a
branch divergence.
- Re-ran the issue-bound tree: all 17 focused contract tests pass in the
hermetic no-country-import environment; whole-repository Ruff format and lint,
focused mypy, architecture Markdown parsing, Towncrier, and diff checks pass.
The shared canonical clone environment has since picked up an uncertified US
package/data combination, so it is no longer a valid runner for this isolated
branch; the hermetic contract suite is unaffected.

## Next

- Push the same-repository branch, open only a draft PR beginning `Fixes #510`,
and verify its live base/head/draft state.
- Freeze the published head for independent review; keep the full docs render
as a disclosed environment-only residual check.
1 change: 1 addition & 0 deletions changelog.d/510.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add typed behavior-input bindings and stable-ID resolution for population data used by future behavioral adapters.
49 changes: 48 additions & 1 deletion docs/methodology/model-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,54 @@ A complete model page should be explicit about four pieces:

For example, a program page should not stop at eligibility. It should say how benefit value is represented, whether household-paid costs are modeled, what data inputs are required, and how the program enters aggregate output concepts.

## Behavior inputs and legal semantics

Legal rules, population construction, and non-legal behavioral mechanics have
different owners:

| Layer | Owns |
|---|---|
| Source systems and Chronicle | Documentary source facts |
| Microcosm | Population construction and measured or latent population flags |
| PolicyEngine | Explicit population-input bindings and, in later stages, non-legal take-up and labor-supply mechanics |
| Axiom and RuleSpec | Only legal rules, concepts, events, and statuses grounded in public documents |

These terms are not interchangeable:

- **Eligibility** is a legal qualifying predicate.
- **Entitlement** is a legal right or calculated amount. It does not establish
application, award, payment, or receipt.
- **Application or claim** is a claimant or administrative event. Microcosm may
carry measured or latent application state; Axiom may receive it only when an
exact public authority makes it legally operative.
- **Award** is an administrative determination. It must not be inferred from
eligibility or receipt.
- **Payment** is a legal amount due, issued, or disbursed.
- **Receipt** is a measured or latent population fact or a PolicyEngine
behavioral outcome. It may lag or differ from legal payment.
- **Simulated non-take-up** is a PolicyEngine-owned behavioral outcome. It is
not ineligibility, denial, loss of entitlement, or an Axiom fact.

Eligibility or a positive static amount alone proves none of application,
award, payment, or receipt.

`BehaviorInputBinding` gives an adapter-local role an entity and column
reference. Roles are local labels, not universal legal or benefit concepts;
bindings do not contain arrays, dataframes, simulations, callables,
probabilities, or arbitrary objects. `BehaviorInputs` is the frozen,
JSON-round-trippable collection of those bindings.

The internal resolver reads only declared columns from loaded
`YearData.entity_data`. It requires a non-null, unique `<entity>_id`, copies
values into series indexed by that stable ID, and preserves nulls. It never
aligns rows positionally, remaps between entities, or requires a column to
appear in a legal-model variable registry.

This boundary only validates and resolves inputs. It adds no behavior or
Belgian formula, `takes_up_*` concept, adapter registry, legal rerun,
`Simulation` field, cache behavior, public `pe.be` export, effect on
`Simulation.run()`, or country and labor-supply numerical change.

## What belongs in generated reference

Generated reference pages should include:
Expand All @@ -70,4 +118,3 @@ Authored methodology pages should focus on model choices:
- what current limitations users should know before interpreting outputs

That is the structure used by the first new US health-cost page.

2 changes: 2 additions & 0 deletions src/policyengine/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
provenance layer.
"""

from .behavior import BehaviorInputBinding as BehaviorInputBinding
from .behavior import BehaviorInputs as BehaviorInputs
from .dataset import Dataset
from .dataset import YearData as YearData
from .dataset import map_to_entity as map_to_entity
Expand Down
108 changes: 108 additions & 0 deletions src/policyengine/core/behavior.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
"""Data-only configuration for adapter-local behavior inputs."""

from typing import Optional

import pandas as pd
from pydantic import BaseModel, ConfigDict, field_validator

from .dataset import YearData

__all__ = ["BehaviorInputBinding", "BehaviorInputs"]


class BehaviorInputBinding(BaseModel):
"""Bind an adapter-local role to one population entity column."""

model_config = ConfigDict(frozen=True, extra="forbid")

role: str
entity: str
column: str


class BehaviorInputs(BaseModel):
"""Immutable population-column bindings for a behavior adapter."""

model_config = ConfigDict(frozen=True, extra="forbid")

bindings: tuple[BehaviorInputBinding, ...]

@field_validator("bindings")
@classmethod
def _roles_must_be_unique(
cls,
bindings: tuple[BehaviorInputBinding, ...],
) -> tuple[BehaviorInputBinding, ...]:
seen: set[str] = set()
duplicates: list[str] = []
for binding in bindings:
if binding.role in seen and binding.role not in duplicates:
duplicates.append(binding.role)
seen.add(binding.role)
if duplicates:
raise ValueError(
"Behavior input roles contain duplicate adapter-local labels: "
f"{duplicates}."
)
return bindings


def _resolve_behavior_inputs(
behavior_inputs: BehaviorInputs,
data: Optional[YearData],
) -> dict[str, pd.Series]:
"""Resolve bindings to copied values indexed only by their entity IDs."""
if data is None:
raise ValueError("Behavior input resolution requires loaded YearData.")

entity_data = data.entity_data
if entity_data is None:
raise ValueError("Behavior input resolution requires loaded YearData.")

resolved: dict[str, pd.Series] = {}
for binding in behavior_inputs.bindings:
if binding.entity not in entity_data:
raise ValueError(
f"Behavior input role {binding.role!r} references missing entity "
f"{binding.entity!r}."
)

entity_table = entity_data[binding.entity]
if entity_table is None:
raise ValueError(
f"Behavior input role {binding.role!r} references entity "
f"{binding.entity!r} without loaded data."
)
table = pd.DataFrame(entity_table)
id_column = f"{binding.entity}_id"
if id_column not in table.columns:
raise ValueError(
f"Behavior input entity {binding.entity!r} is missing required ID "
f"column {id_column!r}."
)

ids = table[id_column].copy(deep=True)
if ids.isna().any():
raise ValueError(
f"Behavior input entity {binding.entity!r} has null values in "
f"required ID column {id_column!r}."
)
duplicate_ids = ids.duplicated(keep=False)
if duplicate_ids.any():
duplicate_values = ids.loc[duplicate_ids].drop_duplicates().tolist()
raise ValueError(
f"Behavior input entity {binding.entity!r} must have unique "
f"{id_column!r} values; duplicates: {duplicate_values}."
)

if binding.column not in table.columns:
raise ValueError(
f"Behavior input role {binding.role!r} references missing column "
f"{binding.column!r} on entity {binding.entity!r}."
)

values = table[binding.column].copy(deep=True)
values.index = pd.Index(ids.array.copy(), name=id_column)
resolved[binding.role] = values

return resolved
Loading
Loading