Skip to content

Refine %catalog context resolution: strict use(), optional list() path #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
oskaresparza merged 1 commit into from
Sep 18, 2026
Merged

Refine %catalog context resolution: strict use(), optional list() path #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 31 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,25 +203,36 @@ custodian, not necessarily a developer. `%ingest help` does the same for `Ingest

`%%catalog` (the cell-magic form) sets the context once, with `use(path)` on its magic line,
then runs every other line of the cell in order under that context, without repeating the full
path on each line. A leading `.` resolves any `path`/`source_path`/`target_path` against the
context (so `data_copy`/`set_tags`/`create_folder`/... all understand it); `use` alone also accepts a bare
path with no dot, once a context exists — `"2027"` and `".2027"` narrow it the same way there.
One or more leading `../` (or a bare `..`) instead walks up that many levels of the context
first, everywhere a relative path is understood, not just in `use` — `"../water_temperature"` is
a sibling of the context, `"../../water_temperature"` a level further up. `use` also makes a live
check that the resolved path actually exists in the catalog, raising if it doesn't, rather than
silently pointing context somewhere later calls would fail against anyway; `use(None)` clears the
context. `get_context()` shows what it currently is — always the full resolved path:
path on each line. `use`'s own `path` must always be a whole, absolute path — unlike every
other verb, it's never resolved against whatever context already exists, and never accepts a
leading `.`/`../`. It does make a live check that `path` actually exists in the catalog first,
raising if it doesn't, rather than silently pointing context somewhere later calls would fail
against anyway; `use(None)` clears the context. Every other verb's own `path`/`source_path`/
`target_path` still resolves against the current context once one is set — with or without a
leading `.` (`"2027"` and `".2027"` mean the same thing) — unless it already starts with
`catalog` (this deployment's one real root source), in which case it's always taken literally
as absolute rather than appended to the context, dot or not. `data_copy`/`data_move`/
`create_view`'s two paths are a further exception: they resolve `source_path`/`target_path`
independently against that same context, so routinely pair a relative one with a genuinely
unrelated absolute one, and still require the dot to mean relative (a bare path there is
always absolute, context or not, `catalog`-prefixed or not). One or more leading `../` (or a
bare `..`) instead walks up that many levels of the context first, everywhere a relative path
is understood except inside `use` itself — `"../water_temperature"` is a sibling of the
context, `"../../water_temperature"` a level further up. `get_context()` shows what the
context currently is — always the full resolved path:

```python
%%catalog use("bwd.reference")
set_tags(".water_temperature", ["reviewed"])
create_folder(".2027") # create_folder still needs the dot — only use() makes it optional

%catalog use("bwd.reference") # back to a path that already exists
%catalog use("2027") # bare, no dot — same as use(".2027"); already created above
%catalog get_context() # -> 'bwd.reference.2027'
%catalog set_tags("../water_temperature", ["archived"]) # ../ works for any verb, not just use()
set_tags("water_temperature", ["reviewed"]) # bare, no dot — same as ".water_temperature"
create_folder(".2027")

%catalog use("bwd.reference.2027") # use() always takes the whole, absolute path
%catalog get_context() # -> 'bwd.reference.2027'
%catalog set_tags("../water_temperature", ["archived"]) # ../ works for any verb except use()
%catalog get_tags("catalog.other_root.assessments") # starts with 'catalog' — absolute, not appended
%catalog data_copy(".water_temperature", "other_root.archive.water_temperature_2027")
# ^ bare, but data_copy/data_move/create_view always
# take a bare target_path/source_path literally
```

`get_wiki(path)`, `get_tags(path)`, `list(path)` (every table/view under `path`, at any depth)
Expand All @@ -230,12 +241,15 @@ to commit or undo. `delete_view(path)`/`delete_table(path)` queue like every oth
unlike the idempotent `DROP ... IF EXISTS` `Catalog.deleteview`/`deletetable` wrap — require
`path` to already exist, and (like `delete_folder`) can never be undone. Every one of these
raises `CatalogOperationError` if `path` doesn't exist; `schema` also requires it to be a table
or view, not a folder:
or view, not a folder. `list`'s `path` may also be omitted (or `""`) to list the current
context itself — raises `CatalogSessionError` if none is set yet:

```python
%catalog get_wiki("bwd.reference.water_temperature")
%catalog get_tags("bwd.reference.water_temperature")
%catalog list("bwd.reference") # -> ['bwd.reference.water_temperature', ...]
%catalog use("bwd.reference")
%catalog list() # same result — path omitted, lists the context itself
%catalog schema("bwd.reference.water_temperature") # -> TableInfo(schema={...}, row_count=...)
%catalog delete_view("bwd.reference.old_view")
```
Expand Down
37 changes: 27 additions & 10 deletions debugger/test_catalog_magic.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,12 @@
"%catalog set_wiki(TEST_FOLDER, \"# Catalog magic smoke test\\n\\nCreated by the %catalog debugger notebook.\")"
]
},
{
"cell_type": "markdown",
"id": "851a3985",
"source": "## 4. Read-only queries answer immediately too\n\n`get_wiki`/`get_tags`/`list`/`schema` are read-only `CatalogSession` methods — no `%catalog\ncommit` involved, same as every other call in this notebook. `get_wiki` reads back the wiki\njust set above; `list` finds no tables/views here since `TEST_FOLDER` only holds folders —\nan empty list, not an error (it would only raise if `TEST_FOLDER` itself didn't exist).\n`list`'s own `path` can also be omitted entirely, to list whatever the current context is —\nthe next cell does exactly that, having just pointed context at `TEST_FOLDER` via `use`. The\ncell after that passes `TEST_ROOT` — a `catalog.`-prefixed path — while context is still\n`TEST_FOLDER`, to prove it's taken literally as absolute rather than appended to it (it\nwould otherwise become `TEST_FOLDER.catalog....`, which doesn't exist, and raise).",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -229,17 +235,28 @@
]
},
{
"cell_type": "markdown",
"id": "59854882",
"cell_type": "code",
"id": "70ce527b",
"source": "%catalog list(TEST_FOLDER) # -> [] (only folders live here — not an error, unlike a typo'd path)",
"metadata": {},
"source": [
"## 4. Read-only queries answer immediately too\n",
"\n",
"`get_wiki`/`get_tags`/`list`/`schema` are read-only `CatalogSession` methods — no `%catalog\n",
"commit` involved, same as every other call in this notebook. `get_wiki` reads back the wiki\n",
"just set above; `list` finds no tables/views here since `TEST_FOLDER` only holds folders —\n",
"an empty list, not an error (it would only raise if `TEST_FOLDER` itself didn't exist)."
]
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"id": "66eb277b",
"source": "%catalog use(TEST_FOLDER)\n%catalog list() # path omitted — lists the current context (TEST_FOLDER) itself; same [] result",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"id": "60e1494f",
"source": "%catalog list(TEST_ROOT) # starts with 'catalog' — always absolute, never appended to\n # the current context (still TEST_FOLDER, per the cell above)",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
Expand Down
61 changes: 48 additions & 13 deletions docs/notebook-facade-for-data-scientists.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,11 @@ behaviour described below is unchanged; there's just no longer a separate
still queues/commits as described throughout this doc. Context has also
grown past what's described below: `use(path)` (also via `%%catalog
use(path)`, the cell-magic form) sets it deliberately — making a live
check that the resolved path exists in the catalog first, unlike
everything else here, and always storing the full resolved path, never a
raw `.`/`..`-prefixed fragment — `get_context()` reads it back, and a
leading `../` (or a bare `..`) on any relative path, not just inside
`use`, walks up that many levels of the context first. `copy`/`move` were
later renamed `datacopy`/`datamove`, matching `Catalog`'s own names instead
of inventing friendlier ones, then renamed again to `data_copy`/`data_move`
for consistency with the rest of the facade's underscored verbs (`Catalog`'s
check that `path` exists in the catalog first, unlike everything else
here — and `get_context()` reads it back. `copy`/`move` were later renamed
`datacopy`/`datamove`, matching `Catalog`'s own names instead of inventing
friendlier ones, then renamed again to `data_copy`/`data_move` for
consistency with the rest of the facade's underscored verbs (`Catalog`'s
own `datacopy`/`datamove` are unchanged — only the `CatalogSession`/
`%catalog` wrapper got the underscore); six more verbs were added — read-only
`get_wiki`, `get_tags`, `list`, `schema` (answered immediately, like
Expand All @@ -42,10 +39,46 @@ for consistency with `delete_folder`/`delete_wiki`). `tag`/`untag` were
similarly renamed `set_tags`/`delete_tags` (matching `set_wiki`/
`delete_wiki`'s pattern), and `set_meta` was removed — the raw
`Catalog.setmeta2wiki`/`getmetafromwiki` are still there for a folder's
wiki Meta Data section, just not wrapped by `CatalogSession` any more. See
`src/eea_datalakehouse/notebook/magics.py`'s module docstring and
wiki Meta Data section, just not wrapped by `CatalogSession` any more.

Relative-path resolution grew, then partly retreated. It first grew past
a leading `.`: a leading `../` (or a bare `..`) on any relative path
walked up that many levels of the context first, and — since requiring a
dot everywhere turned out to be a real papercut in practice (a custodian's
first instinct after `use(...)` was to type a bare short name regardless
of which verb came next) — every single-path verb started accepting a
bare path with no leading `.` at all too, once a context existed, the same
as `use` already did (`data_copy`/`data_move`/`create_view` are the
deliberate exception: they resolve `source_path`/`target_path`
independently against the *same* starting context and routinely pair a
relative one with a genuinely unrelated absolute one in the same call, so
a bare path there still always means absolute, context or not). `use`
itself then reverted the other way: it now only ever accepts a whole,
absolute `path` — never resolved against whatever context already exists,
and never a leading `.`/`../` fragment — so pointing context somewhere
always means saying exactly where, with the same live existence check as
before. Every other verb's own relative-path behaviour (dot-optional,
`../`-aware) is unchanged. `list`'s own `path` became optional on top of
that — omitted (or `""`), it lists the current context itself, raising
`CatalogSessionError` if none is set, rather than making a custodian who's
already `use()`d somewhere repeat that same path right back to `list()`.

The dot-optional rule then grew one more exception of its own: a bare path
that already starts with `catalog` (`_ROOT_SOURCE` in `session.py` — this
deployment's one real top-level source) is always taken literally as
absolute, context or not, rather than getting appended to whatever
context happens to be set. Before this, passing a full `catalog....` path
alongside an already-set context — mixing an absolute path with ordinary
relative use in the same session — silently produced a nonsense
double-nested path (`f"{context}.catalog...."`) unless a custodian
remembered to clear context first; `_ROOT_SOURCE` makes that case
unambiguous instead. `data_copy`/`data_move`/`create_view` don't get this
treatment — a `_ROOT_SOURCE` check can't tell a deliberately relative bare
path (still meant to be appended there) apart from one that just happens
not to start with `_ROOT_SOURCE`, so they keep requiring the dot outright.
See `src/eea_datalakehouse/notebook/magics.py`'s module docstring and
`src/eea_datalakehouse/catalog/session.py`'s `use`/`get_context`/
`_resolve_path` for the current, authoritative behaviour.
`_resolve_path`/`list` for the current, authoritative behaviour.

## The facade's surface, end to end

Expand Down Expand Up @@ -319,8 +352,10 @@ This splits into two sides that live in two different repositories.
open question below with an explicit marker (a leading `.`, e.g.
`session.set_tags(".water_temperature", ...)`), and — critically — `set_context`
itself is not something a data custodian is expected to call directly
(`use`, added later, is the custodian-facing entry point built on the same
mechanism — see "What it exposes" above):
(`use`, added later, is the custodian-facing entry point for setting
context deliberately — see "What it exposes" above — though it takes only
a whole, absolute path, plus a live existence check `set_context` itself
doesn't make):

- **Ordinary use already keeps it current on its own.** Every queueing verb
updates the context from whatever path it just touched (the target's
Expand Down
14 changes: 7 additions & 7 deletions docs/notebooks/catalog_session_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
{
"cell_type": "markdown",
"id": "0f5e1ae8",
"source": "## Quick reference\n\n`%catalog help` (or `%catalog help()`) prints every command as a plain table — name,\nparameters, description — rather than a raw Python signature, since a data custodian\nreading it may not be fluent in Python type-hint syntax. Handy when you don't remember\nan exact parameter name mid-notebook. It works even before `DREMIO_BASE_URL`/\n`DREMIO_TOKEN` are set, since it's answered before a session is built.",
"source": "## Quick reference\n\n`%catalog help` (or `%catalog help()`) renders every command as an HTML table — name,\nparameters, description — rather than a raw Python signature, since a data custodian\nreading it may not be fluent in Python type-hint syntax. Handy when you don't remember\nan exact parameter name mid-notebook. It works even before `DREMIO_BASE_URL`/\n`DREMIO_TOKEN` are set, since it's answered before a session is built.",
"metadata": {}
},
{
Expand Down Expand Up @@ -80,29 +80,29 @@
{
"cell_type": "code",
"id": "7e896ad6",
"source": "%%catalog use(\"bwd.reference\")\nset_tags(\".water_temperature\", [\"archived\"])\ncreate_folder(\".2027\")",
"source": "%%catalog use(\"bwd.reference\")\nset_tags(\"water_temperature\", [\"archived\"])\ncreate_folder(\"2027\")",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "4b6abba9",
"source": "### `use`'s two differences from every other verb\n\n`create_folder` above still needed the leading `.` on `.2027` — every verb's own\n`path`/`source_path`/`target_path` does, so an absolute path can always be passed even\nwith a context already set. `use` alone accepts a bare path too, once a context\nexists — `use(\"2027\")` and `use(\".2027\")` mean the same thing there. `use(None)` clears\nthe context entirely, the same as `set_context(None)`.\n\n`use` also makes a live check: the resolved path must already exist in the catalog, or\nit raises instead of quietly pointing context somewhere later calls would fail against\nanyway. That's why the cell below re-enters `\"bwd.reference.2027\"`, created above,\nrather than a path nothing has created yet.",
"source": "### `use`'s live existence check\n\n`create_folder`/`set_tags` above dropped the leading `.` on `2027`/`water_temperature`\nentirely — once a context exists, every single-path verb's own `path` accepts a bare\nname this way (`\"2027\"` and `\".2027\"` mean the same thing). There are two exceptions:\n\n- `use` itself: its own `path` must always be a whole, absolute path — never resolved\n against whatever context already exists, and never a leading `.`/`../` fragment.\n- `data_copy`/`data_move`/`create_view`: they resolve `source_path`/`target_path`\n independently against that same starting context, so routinely pair a relative one\n with a genuinely unrelated absolute one — a bare path there still always means\n absolute, context or not.\n\nA bare path starting with `catalog` (this deployment's one real root source, e.g.\n`\"catalog.other_root.assessments\"`) is also always taken literally as absolute rather\nthan appended to the context — but that's not an exception to the dot-optional rule\nabove, just a further refinement of what \"bare\" means: without it, passing a full\n`catalog....` path alongside an already-set context would silently produce a nonsense\ndouble-nested path instead of raising or doing the obvious thing.\n\n`use(None)` clears the context entirely, the same as `set_context(None)`.\n\n`use` also makes a live check none of the others do: `path` must already exist in the\ncatalog, or it raises instead of quietly pointing context somewhere later calls would fail\nagainst anyway. That's why the cell below re-enters the full `\"bwd.reference.2027\"`,\ncreated above, rather than a path nothing has created yet.",
"metadata": {}
},
{
"cell_type": "code",
"id": "417250b0",
"source": "%catalog use(\"bwd.reference\") # back to a path that already exists\n%catalog use(\"2027\") # bare, no dot — same as use(\".2027\"); already created above\n%catalog get_context() # -> 'bwd.reference.2027'",
"source": "%catalog use(\"bwd.reference.2027\") # use() always takes the whole, absolute path\n%catalog get_context() # -> 'bwd.reference.2027'",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "9be5a25f",
"source": "### Going up: `../`\n\nOne or more leading `../` (or a bare `..`) walks up that many levels of the context\nfirst, then resolves whatever's left against the result — and unlike the dot-optional\nshortcut above, this works for *every* relative path, not just `use`'s.",
"source": "### Going up: `../`\n\nOne or more leading `../` (or a bare `..`) walks up that many levels of the context\nfirst, then resolves whatever's left against the result — this works for every relative\npath except `use`'s own (always a whole, absolute path — see above), including\n`data_copy`/`data_move`/`create_view`'s own two paths (the one place among the rest a\nbare path with no dot at all still means \"take it literally as absolute\").",
"metadata": {}
},
{
Expand All @@ -116,7 +116,7 @@
{
"cell_type": "markdown",
"id": "6cbe67f0",
"source": "## Read-only queries: `get_wiki`, `get_tags`, `list`, `schema`\n\nThese answer immediately too, like `get_context()` above — nothing to commit or undo.\nAll four raise `CatalogOperationError` if `path` doesn't exist; `schema` also requires a\ntable or view, not a folder.",
"source": "## Read-only queries: `get_wiki`, `get_tags`, `list`, `schema`\n\nThese answer immediately too, like `get_context()` above — nothing to commit or undo.\nAll four raise `CatalogOperationError` if `path` doesn't exist; `schema` also requires a\ntable or view, not a folder. `list`'s `path` may also be omitted (or `\"\"`) to list the\ncurrent context itself — raises `CatalogSessionError` if none is set yet.",
"metadata": {}
},
{
Expand All @@ -138,7 +138,7 @@
{
"cell_type": "code",
"id": "91156d7b",
"source": "%catalog list(\"bwd.reference\") # every table/view under bwd.reference\n%catalog schema(\"bwd.reference.water_temperature\") # column types + row count, no rows fetched",
"source": "%catalog list(\"bwd.reference\") # every table/view under bwd.reference\n%catalog list() # same thing — path omitted, context is 'bwd.reference'\n%catalog schema(\"bwd.reference.water_temperature\") # column types + row count, no rows fetched",
"metadata": {},
"execution_count": null,
"outputs": []
Expand Down
Loading
Loading