Skip to content

Add country-aware source storage and NZ conventions - #208

Merged
MaxGhenis merged 3 commits into
mainfrom
nz-source-conventions-175
Aug 30, 2026
Merged

Add country-aware source storage and NZ conventions#208
MaxGhenis merged 3 commits into
mainfrom
nz-source-conventions-175

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements the code and documentation portion of #175, the prerequisite for
New Zealand source packages in #176. This does not close #175 because its
credentialed UK object copy and manifests-only sweep remain separate.

  • Infer raw/nz and raw/uk from package publisher directories; apply the same
    country convention to new derived artifacts. Preserve legacy US paths.
  • Refuse raw publication when a manifest-recorded key differs from the
    country-aware key, before invoking an upload.
  • Document New Zealand's ending-year tax label and the four geography and rent
    aggregation rulings. No closed-set vocabulary or schema changes.
  • Add regression tests for mixed-country scans, legacy US paths, unrelated
    ancestor directory names, packages named packages, and refusal of stale or wrong-country keys before
    a fetch can overwrite cached bytes or metadata.
  • Correct the original NZ source conventions: R2 country layout, tax-year labelling, geography rulings #175 quartile guidance: Eurostat Eurostat pilot calibration packages: BE, DE, FR #168 already added
    aggregation: quantile for publisher cut-points with explicit quantile-code
    constraints. No new vocabulary is required.

Verification

  • uv run pytest -q tests/test_chronicle_artifacts.py tests/test_chronicle_core.py tests/test_chronicle_consumer_contract.py tests/test_policyengine_chronicle_imports.py tests/test_policyengine_chronicle_schema.py: 88 passed.
  • uv run ruff check: passed.
  • git diff --check: passed.
  • Full uv run pytest -q: interrupted after 5 minutes and 50 passing tests
    while the bundle build remained CPU-active; no failure had been reported.
    The full suite is not claimed green.

Separate credentialed remainder

The tracked UK manifest census contains 82 artifacts across 82 manifests,
totalling 82,837,846 declared bytes. All 82 still record legacy raw/<publisher>
keys; none records raw/uk yet. This PR changes no UK manifest and deletes no
object.

The current Wrangler OAuth session can list ledger-raw and ledger-derived.
Wrangler's object commands expose get, put, and delete, but not server-side
copy. Cloudflare supports CopyObject through its
S3-compatible API; no
PolicyEngine R2 S3 credential is available in the agent-readable secret store.
The separate migration needs that credential, a paginated remote object
census, verified copy-only operations, then the manifest sweep and full UK
bundle acceptance check. Existing UK keys must remain through the soak period.

Route new raw and derived artifacts through country-aware publisher prefixes, refuse stale manifest keys, and record NZ period/geography rulings. Preserve legacy US paths and leave UK object copying and manifest migration to a separate credentialed change.
Restrict publisher inference to canonical package roots, preserve cached bytes on rejected routes, and correct the stale quartile ruling using the existing quantile contract. Targeted schema, consumer, artifact, core, and import checks: 83 passed.
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Independent review of 544b26e completed: ledger-contract PASS and ledger-boundary PASS. The ancestor-name routing, pre-write refusal, packages-named-packages, and quantile-documentation findings are resolved. The focused 88 tests pass, and both full Chronicle CI checks are green. This lands only the source-storage conventions; #175 remains open for the separate credentialed UK object-copy/manifest migration.

@MaxGhenis
MaxGhenis marked this pull request as ready for review August 30, 2026 01:18
@MaxGhenis
MaxGhenis merged commit 10597ae into main Aug 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NZ source conventions: R2 country layout, tax-year labelling, geography rulings

1 participant