Skip to content

docs: specify R2 collector handoff and runtime #1232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
klaasnicolaas merged 3 commits into from
Sep 14, 2026
Merged

docs: specify R2 collector handoff and runtime #1232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d97a677
docs: specify R2 producer handoff and runtime
klaasnicolaas Sep 14, 2026
c2fc334
docs: write import contract and technology stack in English
klaasnicolaas Sep 14, 2026
3669703
docs: link pilot PRs and clarify Content-MD5 encoding
klaasnicolaas Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions docs/development/data-foundation-stack.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
# Techstack voor de eerste importketen
# Technology stack for the first import pipeline

Status: eenvoudige uitvoeringskeuzes bij [#1214](https://github.com/NIPKaart/core/issues/1214). Zie de [leveringsafspraak](data-import-contract.md) en [werkpakketten](data-foundation-delivery.md). Er is nog geen live producer/core-keten of bucket ingericht.
Status: implementation choices for [#1214](https://github.com/NIPKaart/core/issues/1214). See the [delivery contract](data-import-contract.md) and [work packages](data-foundation-delivery.md). The live Amsterdam export and manually reviewed core import were accepted on 14 September 2026; [disabled-parking #783](https://github.com/NIPKaart/disabled-parking/pull/783) and [core #1222](https://github.com/NIPKaart/core/pull/1222) are merged. The private bucket and automatic execution have not been provisioned.

| Onderdeel | Keuze voor de eerste stap |
| Component | Initial choice |
| --- | --- |
| Core | Bestaande Laravel 13-applicatie, PHP 8.4 baseline en PostgreSQL/PostGIS. |
| Beheer | Bestaande React/Inertia-interface voor vergelijken en beoordelen; geen afzonderlijke importapp. |
| Producer | `disabled-parking`, bestaande Python >=3.11-omgeving en universele bronpackage. Bronpackageversies worden bij de noodzakelijke fix gecontroleerd. |
| Dependencybeheer | uv en `uv.lock`; installatie met `uv sync --locked`, daarna `uv run ...`. Universele packages houden hun eigen tooling. |
| Mapping | Kleine Python-dataclass en expliciete mapping voor één bron. |
| Bestand | Eén begrensd UTF-8 JSON-document, geschreven via tijdelijk bestand en atomische vervanging. |
| Validatie | Expliciete typen en inhoudscontroles aan beide kanten. Opis uit #1222 is beschikbaar als een klein gedeeld schema nuttig blijkt; geen schemarelease-infrastructuur vereist. |
| Tests | Bestaande Python unittest/pre-commit-checks en core Pest. Kleine mappingvoorbeelden; geen gemeentelijke netwerken in gewone CI. |
| Eerste overdracht | Lokaal bestand naar dezelfde core-intakeservice die later bucketbestanden verwerkt. |
| Automatische overdracht | Private bucket; R2 is kandidaat. Provider, regio, kosten, volledigheidsmechanisme en scoped rechten in #1217. |
| Uitvoering later | Eén geplande eindige producentuitvoering en core scheduler/queue. Hosting en ophaalritme pas kiezen bij #775/#1217. |
| Core | Existing Laravel 13 application, PHP 8.4 baseline and PostgreSQL/PostGIS. |
| Administration | Existing React/Inertia interface for comparison and review; no separate import application. |
| Producer | `disabled-parking`, its existing Python >=3.12 environment and reusable source package. Check source package versions when a required fix is made. |
| Dependency management | uv and `uv.lock`; install with `uv sync --locked`, then use `uv run ...`. Reusable packages retain their own tooling. |
| Mapping | Small Python dataclass and explicit mapping for one source. |
| File | One bounded UTF-8 JSON document, written through a temporary file and atomic replacement. |
| Validation | Explicit type and content checks on both sides. The Opis/schema prototype was removed in [core #1222](https://github.com/NIPKaart/core/pull/1222); one intake service handles the pilot format. |
| Tests | Existing Python unittest/pre-commit checks and core Pest. Small mapping examples; no municipal network calls in regular CI. |
| Initial handoff | Local file passed to the same core intake service that will later process bucket objects. |
| Automatic handoff | Private Cloudflare R2 bucket. Region, costs, access rights and staging validation belong to #1217; no resources have been provisioned. |
| Scheduled execution | One persistent Docker container per producer repository on an owner-managed Linux host or cloud VM. An internal scheduler runs finite commands serially; no container per municipality. Core uses its own scheduler/queue. Implementation in #775/#1217. |

De uv-migratie en SQL-runtimeverwijdering zijn gemerged in disabled-parking #779 en #780. #781 bevat één klein Hamburg-voorbeeld; dat is nog geen live aansluiting. De tijdelijke export wordt in #774 vervangen door de afgesproken pilotroute. Core bevat geen Python-code of bronclients.
The uv migration and SQL runtime removal were merged in [disabled-parking #779](https://github.com/NIPKaart/disabled-parking/pull/779) and [#780](https://github.com/NIPKaart/disabled-parking/pull/780). The temporary Hamburg export from [#781](https://github.com/NIPKaart/disabled-parking/pull/781) was replaced by the live Amsterdam route in [#783](https://github.com/NIPKaart/disabled-parking/pull/783). [Core #1222](https://github.com/NIPKaart/core/pull/1222) provides the corresponding review and publication flow. Core contains no Python code or source clients.

Gebruik bestaande standaardbibliotheken en dependencies waar die voldoen. Geen nieuwe broker, workerframework, lokale SQLite-planningsdatabase, verplichte sequences, JSONL of apart manifest voor de eerste bron. Voeg een component pas toe wanneer de werkende keten die aantoonbaar nodig heeft.
Use standard libraries and existing dependencies where they suffice. The first source needs no new broker, worker framework, local SQLite scheduling database, mandatory sequences, JSONL or separate manifest. Add a component only when the working pipeline demonstrates a need for it.

De producent krijgt uitsluitend de noodzakelijke rechten op zijn eigen private bucketlocatie. Core krijgt aparte leesrechten en beslist over publicatie. Geen databasecredentials of coretoken in de producer. Het overdrachtsmechanisme moet complete, unieke leveringen garanderen; daarvoor is niet vooraf één bepaalde S3-versioningimplementatie voorgeschreven.
The producer should receive only the necessary permissions for its private bucket destination. Core receives separate read access and decides publication. The producer holds no database credentials or core token. The handoff must guarantee complete, unique deliveries without prescribing a particular S3 versioning implementation upfront. Actual credential boundaries must be verified in #1217.

Eerst handmatig de hele keten bewijzen, daarna dezelfde bestanden automatisch overdragen. Bij groei meten we looptijd, bronlimieten, bestandsomvang en beheerwerk voordat we extra processen of gedeelde frameworks toevoegen. Offstreet volgt met eigen inhoudelijke afspraken; algemene garagebezetting bewijst geen beschikbare gehandicaptenparkeerplaats.
First prove the entire pipeline manually, then automate delivery of the same files. As usage grows, measure runtime, source limits, file size and maintenance effort before adding processes or shared frameworks. Offstreet follows with its own data semantics; general garage occupancy does not establish the availability of an accessible parking space.
Loading