Add a notebook-facing facade: CatalogSession/IngestSession + %catalog/%ingest

README.md

@@ -36,7 +36,7 @@ pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"
 ```
 
 # staging's latest release (early access) — pin to the tag the "staging" badge above shows
-pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.11-staging"
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12-staging"
 ```
 
 ## Usage
@@ -238,10 +238,10 @@ To pin to one specific release instead, use the exact tag the live badges under
 [Releasing a new version](#releasing-a-new-version)):
 
 ```python
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.11"
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12"
 
 # staging's latest release (early access)
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.11-staging"
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12-staging"
 ```
 
 Use the `%pip` magic rather than `!pip` — it installs into the kernel the

docs/notebooks/catalog_session_example.ipynb

@@ -0,0 +1,156 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# `%catalog` — queue, commit, rollback\n",
+    "\n",
+    "A worked example of the `%catalog` magic from `eea_datalakehouse.notebook.magics` — see\n",
+    "`docs/notebook-facade-for-data-scientists.md` for the design behind it.\n",
+    "\n",
+    "**Before running this for real:** set `DREMIO_BASE_URL`, `DREMIO_TOKEN` (and `DREMIO_USERNAME`\n",
+    "if you'll queue `copy`/`move`) in the kernel environment — the same variables\n",
+    "`debugger/debug_run.py` uses. Install the extra this needs once: `pip install \"EEADataLakehouse[notebook]\"`.\n",
+    "\n",
+    "Nothing below invents new vocabulary: every call after `%catalog` is a real\n",
+    "`CatalogSession` method (`src/eea_datalakehouse/catalog/session.py`) — this notebook is a tour of\n",
+    "that class, not a separate API."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "import eea_datalakehouse.notebook  # registers %catalog/%ingest — no %load_ext needed"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f469f7f5",
+   "source": "`%load_ext eea_datalakehouse.notebook.magics` still works too, and is safe to run either\nbefore or after the import above — whichever runs first registers the magics, the other is\na no-op (see `magics.load_ipython_extension`'s docstring).",
+   "metadata": {}
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Queue a few steps\n",
+    "\n",
+    "Each cell only **queues** an operation — nothing reaches Dremio yet. The same one\n",
+    "`CatalogSession` is reused across every `%catalog` cell in this kernel (see \"Session\n",
+    "context lives in the Python process\" in the design doc), so calls in different cells\n",
+    "accumulate on the same batch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%catalog copy(\"bwd.draft.raw_2026\", \"bwd.reference.water_temperature\", overwrite=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d1f58114",
+   "source": "%catalog tag(\".water_temperature\", [\"reviewed\", \"2026\"])",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%catalog tag(\"bwd.reference.water_temperature\", [\"reviewed\", \"2026\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%catalog set_meta(\"bwd.reference\", tags=[\n",
+    "    {\"tag_name\": \"owner\", \"tag_value\": \"bathing-water-team\", \"tag_title\": \"Owner\"},\n",
+    "], overwrite=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Commit — all-or-nothing\n",
+    "\n",
+    "`commit()` runs every queued step in order. `retry=True` re-attempts a step that hits a\n",
+    "Dremio engine still warming up (`EngineStartingError`) a few times before giving up — any\n",
+    "other error rolls back immediately regardless. Either way the queue is empty afterwards,\n",
+    "whether this cell succeeds or raises."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%catalog commit(retry=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`commit` with no parentheses also works, as a shorthand:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%catalog commit"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## What a failed commit looks like\n",
+    "\n",
+    "If a step in the batch fails, `%catalog` prints a short message instead of a full\n",
+    "traceback (see the design doc's \"Exceptions translated at the boundary\") — for example,\n",
+    "queuing a step against a target that already exists without `overwrite=True`:\n",
+    "\n",
+    "```\n",
+    "%catalog copy(\"bwd.draft.raw_2026\", \"bwd.reference.water_temperature\")\n",
+    "%catalog commit\n",
+    "# -> catalog error: commit failed at step 1/1 (copy 'bwd.draft.raw_2026' -> ...):\n",
+    "#    target 'bwd.reference.water_temperature' already exists — pass overwrite=True ...\n",
+    "#    Everything before it was rolled back.\n",
+    "```\n",
+    "\n",
+    "A batch mixing a reversible step with an `overwrite=True` step is explicit about what it\n",
+    "could *not* undo rather than pretending the rollback was clean — see `CatalogCommitError`'s\n",
+    "docstring, and \"Must be all-or-nothing\" in the design doc, for exactly which verbs that\n",
+    "applies to (`delete_folder`, and `copy`/`move` when `overwrite=True`)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/notebooks/ingest_session_example.ipynb

@@ -0,0 +1,144 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# `%ingest` — queue folder ingests, commit as one batch\n",
+    "\n",
+    "A worked example of the `%ingest` magic from `eea_datalakehouse.notebook.magics` — see\n",
+    "`docs/notebook-facade-for-data-scientists.md` for the design behind it.\n",
+    "\n",
+    "**Before running this for real:** the kernel environment needs whatever\n",
+    "`eea_datalakehouse.dds_ingestion.credentials.load_creds`/`load_base_url` expect (same as\n",
+    "importing `FolderIngest` directly today). Install the extra this needs once:\n",
+    "`pip install \"EEADataLakehouse[notebook]\"`.\n",
+    "\n",
+    "**This is a separate session from `%catalog`'s, on purpose.** A catalog operation can't run\n",
+    "before its target has actually been ingested, so the two were never one atomic batch — see\n",
+    "\"Two sessions, not one\" in the design doc. Ingest and commit here first; only afterwards, in\n",
+    "a fresh set of `%catalog` cells (see `catalog_session_example.ipynb`), tag or copy what just\n",
+    "landed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "import eea_datalakehouse.notebook  # registers %catalog/%ingest — no %load_ext needed"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d9463089",
+   "source": "`%load_ext eea_datalakehouse.notebook.magics` still works too, and is safe to run either\nbefore or after the import above — whichever runs first registers the magics, the other is\na no-op (see `magics.load_ipython_extension`'s docstring).",
+   "metadata": {}
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Queue one or more folder ingests\n",
+    "\n",
+    "Each call only records the intent — `folder`/`target_catalog_path`/`data_format` plus\n",
+    "anything else `FolderIngest` accepts (`intent`, `table_name`, `sub_path`, ...). Nothing\n",
+    "uploads until `commit()`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%ingest ingest(folder=\"./bw_2026\", target_catalog_path=\"bwd.reference\", data_format=\"parquet\", intent=\"read_only\", table_name=\"water_temperature\", sub_path=\"2026\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A second ingest into a different table, queued on the *same* session — both run when the\n",
+    "cell below commits:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%ingest ingest(folder=\"./bw_stations_2026\", target_catalog_path=\"bwd.reference\", data_format=\"parquet\", table_name=\"stations\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Commit — runs every queued ingest in order\n",
+    "\n",
+    "Unlike `%catalog`'s `commit()`, this one **cannot roll back** an ingest that already\n",
+    "succeeded — see the design doc and `docs/read-only-ingest-client-plan.md` for why (deleting\n",
+    "an ingested table's backing data needs a server-side operation this package does not have\n",
+    "yet). If the second ingest above were to fail, the first one's table stays exactly as\n",
+    "ingested — `retry=True` only covers resuming a single failed ingest\n",
+    "(`FolderIngest.retry`'s own resume-the-load-step / re-upload-under-a-new-session logic),\n",
+    "not undoing an earlier one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%ingest commit(retry=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## What a partial failure looks like\n",
+    "\n",
+    "```\n",
+    "%ingest commit\n",
+    "# -> ingest error: ingest './bw_stations_2026' -> 'bwd.reference' failed (...); 1 earlier\n",
+    "#    ingest(s) in this batch already committed and CANNOT be undone\n",
+    "```\n",
+    "\n",
+    "That message is deliberately explicit about what already landed — check\n",
+    "`IngestCommitError.succeeded` (or just the catalog) before deciding what to do about the one\n",
+    "that failed, rather than assuming the whole batch was undone."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Now the catalog side, in a fresh session\n",
+    "\n",
+    "Once the ingest above has actually committed, a *separate* `%catalog` batch can tag or copy\n",
+    "what just landed — see `catalog_session_example.ipynb`:\n",
+    "\n",
+    "```\n",
+    "%catalog tag(\"bwd.reference.water_temperature\", [\"reviewed\", \"2026\"])\n",
+    "%catalog commit\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "EEADataLakehouse"
-version = "0.1.11"
+version = "0.1.12"
 description = "Data preparation and validation utilities for the EEA data lakehouse pipeline."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -30,13 +30,17 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
+notebook = [
+    "ipython>=8.18",     # eea_datalakehouse.notebook's %catalog/%ingest magics
+]
 dev = [
     "pytest>=7",
     "ruff>=0.15",
     "mypy>=2.1",
     "respx>=0.23",       # mocks httpx for the ingestion test suite
     "types-tqdm",
     "ipykernel>=6.29",   # run/debug notebooks against this environment
+    "EEADataLakehouse[notebook]",
 ]
 
 [project.urls]

src/eea_datalakehouse/catalog/__init__.py

@@ -30,19 +30,20 @@
     deleteview,
     deletewiki,
     draft2version,
+    getmetafromwiki,
     gettableitemsfrom,
     gettablesfrom,
     gettagsfrom,
     getwikifrom,
     publishversion,
-    getmetafromwiki,
     retry_pending,
     setmeta2wiki,
     settagsto,
     setwikito,
     table2view,
 )
 from .rest import CatalogRestClient
+from .session import CatalogCommitError, CatalogSession, CatalogSessionError, CommitReport
 from .sql import (
     FLIGHT_LOCATION_ENV_VAR,
     TRANSPORT_ENV_VAR,
@@ -57,8 +58,12 @@
     "FLIGHT_LOCATION_ENV_VAR",
     "TRANSPORT_ENV_VAR",
     "Catalog",
+    "CatalogCommitError",
     "CatalogOperationError",
     "CatalogRestClient",
+    "CatalogSession",
+    "CatalogSessionError",
+    "CommitReport",
     "EngineStartingError",
     "FlightSqlExecutor",
     "RestSqlExecutor",