docs: specify R2 collector handoff and runtime - #1232
Merged
Merged
Conversation
klaasnicolaas
added
the
documentation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new documentation introduces ambiguous/unlinked PR/issue references and omits an important Content-MD5 encoding detail that could lead to incorrect S3/R2 implementations.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR updates the import-chain documentation to record the agreed Cloudflare R2 handoff semantics and the “single persistent producer container with internal scheduler” runtime, and to correct stale statements about the now-merged Amsterdam pilot.
Changes:
- Document the selected R2 object naming, integrity checks (MD5 + SHA-256), and replay/conflict handling expectations.
- Document the selected producer runtime model (single persistent container, serial finite commands, retry with persisted pending bytes).
- Update the foundation stack doc to reflect the merged Amsterdam export + reviewed core import and remove stale Opis/schema statements.
File summaries
| File | Description |
|---|---|
| docs/development/data-import-contract.md | Adds the concrete R2 handoff + producer runtime agreement details for the next import milestone. |
| docs/development/data-foundation-stack.md | Updates status and stack decisions to reflect the merged Amsterdam pilot and removes stale Opis/schema wording. |
Review details
Suppressed comments (2)
docs/development/data-foundation-stack.md:13
- De verwijzing "#1222" is hier ambigu (issue vs PR) en niet gelinkt, terwijl elders in de importdocs expliciet naar "PR #1222" wordt gelinkt. Dit voorkomt verwarring over waar de Opis/schema-proef precies is verwijderd.
| Validatie | Expliciete typen en inhoudscontroles aan beide kanten. De Opis/schema-proef is in #1222 verwijderd; één intakeservice verwerkt het pilotformaat. |
docs/development/data-import-contract.md:101
- De tekst verwijst naar core-issues (#1216 en #1217) als platte tekst. Omdat dit document al GitHub-links gebruikt (bijv. #1214), helpt het om deze issues ook te linken zodat implementatie/acceptatiecriteria snel terug te vinden zijn.
De nog te implementeren consumer in core #1216 ontdekt complete objecten binnen de toegestane prefix, begrenst downloads op 32 MiB en controleert de SHA-256, bestandsinhoud en overeenkomst tussen objectnaam, dataset en delivery-ID vóór dezelfde intake. Metadata is geen zelfstandige vertrouwensbron. Het bestaande intakebeleid voor conflicten, herhaling, volgorde en beoordeling blijft gelden; ontvangst publiceert niets automatisch.
Eén permanent draaiende producercontainer op een eigen Linux-host voert eindige opdrachten achter elkaar uit. De standaardwachttijd na succes is 24 uur; na fouten maximaal vijf minuten. Een volume bewaart `pending.json` totdat de upload is bevestigd en bewaart daarna `last.json`. Dezelfde bytes worden na herstart opnieuw aangeboden voordat nieuwe gegevens worden opgehaald. De definitieve regio, bewaartermijn langer dan de toegestane core-uitval, budget en daadwerkelijk geteste toegangsrechten blijven onderdeel van #1217. De implementatie en lokale tests vervangen geen herstelproef tegen echte staging-R2 en core.
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
klaasnicolaas
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Record the selected private R2 handoff and single-container collector runtime for the next import milestone. Specify object names, checksums, replay/conflict handling and the still-to-be-built core consumer, and correct stale descriptions of the merged Amsterdam pilot.
Companion to NIPKaart/disabled-parking#787. Related to #1217 and #1216; neither is closed by this documentation change. Region, retention/budget, actual access rights and recovery against isolated R2/core remain open acceptance. No Python code, core runtime changes or provisioning are included.
Validation: compared against the collector implementation and issue handoff;
git diff --checkpasses. Documentation only, so no application tests were run.