Skip to content

Add a notebook-facing facade: CatalogSession/IngestSession + %catalog/%ingest - #48

Merged
oskaresparza merged 5 commits into
stagingfrom
development
Sep 7, 2026
Merged

Add a notebook-facing facade: CatalogSession/IngestSession + %catalog/%ingest#48
oskaresparza merged 5 commits into
stagingfrom
development

Conversation

@oskaresparza

Copy link
Copy Markdown
Collaborator

Summary

  • Add a queue-then-commit facade over the developer-oriented catalog/ingestion APIs, aimed at data custodians in JupyterLab — see docs/notebook-facade-for-data-scientists.md for the full design discussion.
  • CatalogSession: queue copy/move/tag/untag/set_wiki/delete_wiki/set_meta/create_folder/delete_folder, then commit(retry=True) all-or-nothing, with real rollback via compensating actions (except delete_folder and overwrite=True copies/moves, explicitly flagged as unresolvable rather than silently claimed clean).
  • IngestSession: queue FolderIngest runs, commit(retry=True) in order — no rollback by design (documented gap), reports what already landed on a partial failure instead of pretending it can be undone.
  • idempotency_key fully encapsulated in both — never a parameter a custodian sees.
  • CatalogSession context: a leading . on a path resolves against a "current" prefix kept up to date automatically from ordinary use — no set_context() call needed for normal sequential use.
  • %catalog/%ingest (eea_datalakehouse.notebook.magics): thin dispatch onto one session per kernel, friendly error printing instead of tracebacks. Auto-registered by import eea_datalakehouse.notebook — no %load_ext needed, though it still works.
  • A Jupyter Comm channel (eea_datalakehouse.catalog_context) lets an integrated frontend push a selected catalog-tree leaf's path into the context invisibly — the receiving half of a future JupyterLab extension integration (the emitting half lives in the separate eeadata/EEALakeHouse repo).
  • Adds a notebook extra (ipython) and two worked-example notebooks under docs/notebooks/.

Test plan

  • 285 tests passing
  • ruff clean on all touched files
  • mypy clean on the new session modules (the notebook/ package hits a pre-existing numpy-stub/Python-version mismatch in this environment, confirmed to pre-date this work)

🤖 Generated with Claude Code

oskaresparza and others added 5 commits August 27, 2026 13:54
…/%ingest

A queue-then-commit layer over the developer-oriented catalog/ingestion
APIs, aimed at data custodians in JupyterLab rather than application
developers — see docs/notebook-facade-for-data-scientists.md for the full
design discussion this implements.

- CatalogSession: queue copy/move/tag/untag/set_wiki/delete_wiki/set_meta/
  create_folder/delete_folder, then commit(retry=True) all-or-nothing —
  every reversible verb rolls back on failure via a real compensating
  action; delete_folder and overwrite=True copies/moves are explicitly
  flagged as unresolvable rather than silently claimed clean.
- IngestSession: queue FolderIngest runs, commit(retry=True) in order — no
  rollback, by design (no delete-with-backing-data op exists yet; see
  docs/read-only-ingest-client-plan.md), so a partial failure reports what
  already landed instead of pretending it can be undone.
- idempotency_key is fully encapsulated in both — never a parameter a
  custodian sees or passes.
- CatalogSession context: a leading '.' on a path resolves against a
  "current" prefix that ordinary use already keeps up to date on its own
  (no set_context() call needed for normal sequential use).
- %catalog/%ingest (eea_datalakehouse.notebook.magics): thin dispatch onto
  one session per kernel, friendly error printing instead of tracebacks.
  Registered automatically by `import eea_datalakehouse.notebook` (no
  %load_ext needed, though it still works and is safe to combine).
- A Jupyter Comm channel (eea_datalakehouse.catalog_context) lets an
  integrated frontend push a selected catalog-tree leaf's path into the
  context invisibly — the receiving half of a future JupyterLab extension
  integration; the emitting half lives in the separate eeadata/EEALakeHouse
  repo and isn't built here.

Adds a `notebook` extra (ipython) and two worked-example notebooks under
docs/notebooks/. 285 tests passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a notebook-facing facade: CatalogSession/IngestSession + %catalog/%ingest
@oskaresparza
oskaresparza merged commit a032144 into staging Sep 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant