Add a notebook-facing facade: CatalogSession/IngestSession + %catalog/%ingest - #48
Merged
Conversation
…/%ingest A queue-then-commit layer over the developer-oriented catalog/ingestion APIs, aimed at data custodians in JupyterLab rather than application developers — see docs/notebook-facade-for-data-scientists.md for the full design discussion this implements. - CatalogSession: queue copy/move/tag/untag/set_wiki/delete_wiki/set_meta/ create_folder/delete_folder, then commit(retry=True) all-or-nothing — every reversible verb rolls back on failure via a real compensating action; delete_folder and overwrite=True copies/moves are explicitly flagged as unresolvable rather than silently claimed clean. - IngestSession: queue FolderIngest runs, commit(retry=True) in order — no rollback, by design (no delete-with-backing-data op exists yet; see docs/read-only-ingest-client-plan.md), so a partial failure reports what already landed instead of pretending it can be undone. - idempotency_key is fully encapsulated in both — never a parameter a custodian sees or passes. - CatalogSession context: a leading '.' on a path resolves against a "current" prefix that ordinary use already keeps up to date on its own (no set_context() call needed for normal sequential use). - %catalog/%ingest (eea_datalakehouse.notebook.magics): thin dispatch onto one session per kernel, friendly error printing instead of tracebacks. Registered automatically by `import eea_datalakehouse.notebook` (no %load_ext needed, though it still works and is safe to combine). - A Jupyter Comm channel (eea_datalakehouse.catalog_context) lets an integrated frontend push a selected catalog-tree leaf's path into the context invisibly — the receiving half of a future JupyterLab extension integration; the emitting half lives in the separate eeadata/EEALakeHouse repo and isn't built here. Adds a `notebook` extra (ipython) and two worked-example notebooks under docs/notebooks/. 285 tests passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a notebook-facing facade: CatalogSession/IngestSession + %catalog/%ingest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/notebook-facade-for-data-scientists.mdfor the full design discussion.CatalogSession: queuecopy/move/tag/untag/set_wiki/delete_wiki/set_meta/create_folder/delete_folder, thencommit(retry=True)all-or-nothing, with real rollback via compensating actions (exceptdelete_folderandoverwrite=Truecopies/moves, explicitly flagged as unresolvable rather than silently claimed clean).IngestSession: queueFolderIngestruns,commit(retry=True)in order — no rollback by design (documented gap), reports what already landed on a partial failure instead of pretending it can be undone.idempotency_keyfully encapsulated in both — never a parameter a custodian sees.CatalogSessioncontext: a leading.on a path resolves against a "current" prefix kept up to date automatically from ordinary use — noset_context()call needed for normal sequential use.%catalog/%ingest(eea_datalakehouse.notebook.magics): thin dispatch onto one session per kernel, friendly error printing instead of tracebacks. Auto-registered byimport eea_datalakehouse.notebook— no%load_extneeded, though it still works.eea_datalakehouse.catalog_context) lets an integrated frontend push a selected catalog-tree leaf's path into the context invisibly — the receiving half of a future JupyterLab extension integration (the emitting half lives in the separateeeadata/EEALakeHouserepo).notebookextra (ipython) and two worked-example notebooks underdocs/notebooks/.Test plan
notebook/package hits a pre-existing numpy-stub/Python-version mismatch in this environment, confirmed to pre-date this work)🤖 Generated with Claude Code