Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 41 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ examples. Install the extra this needs once: `pip install "EEADataLakehouse[note
import eea_datalakehouse.notebook # registers %catalog/%ingest — no %load_ext needed

%catalog data_copy("draft.raw_2026", "bwd.reference.water_temperature")
%catalog set_tags(".water_temperature", ["reviewed"])
%catalog set_tags("bwd.reference.water_temperature", ["reviewed"])

%ingest ingest(folder="./bw_2026", target_catalog_path="bwd.reference",
data_format="parquet", table_name="water_temperature")
Expand All @@ -201,38 +201,56 @@ parameters, description — rather than a raw Python signature, since its audien
custodian, not necessarily a developer. `%ingest help` does the same for `IngestSession`'s
(fewer) methods — handy when you don't remember an exact parameter name mid-notebook.

A freshly created session already has a context — it starts at `"catalog"` (the catalog
root), not empty — so a relative path works even before the first `use()` call.
`use(None)`/`use("")` reset it back to `"catalog"` the same way; `set_context(None)` (the
lower-level primitive `use` wraps, not normally called directly) is the one way left to
clear it to no context at all.

`%%catalog` (the cell-magic form) sets the context once, with `use(path)` on its magic line,
then runs every other line of the cell in order under that context, without repeating the full
path on each line. `use`'s own `path` must always be a whole, absolute path — unlike every
other verb, it's never resolved against whatever context already exists, and never accepts a
leading `.`/`../`. It does make a live check that `path` actually exists in the catalog first,
raising if it doesn't, rather than silently pointing context somewhere later calls would fail
against anyway; `use(None)` clears the context. Every other verb's own `path`/`source_path`/
`target_path` still resolves against the current context once one is set — with or without a
leading `.` (`"2027"` and `".2027"` mean the same thing) — unless it already starts with
`catalog` (this deployment's one real root source), in which case it's always taken literally
as absolute rather than appended to the context, dot or not. `data_copy`/`data_move`/
`create_view`'s two paths are a further exception: they resolve `source_path`/`target_path`
independently against that same context, so routinely pair a relative one with a genuinely
unrelated absolute one, and still require the dot to mean relative (a bare path there is
always absolute, context or not, `catalog`-prefixed or not). One or more leading `../` (or a
bare `..`) instead walks up that many levels of the context first, everywhere a relative path
is understood except inside `use` itself — `"../water_temperature"` is a sibling of the
context, `"../../water_temperature"` a level further up. `get_context()` shows what the
context currently is — always the full resolved path:
path on each line. `use` is the *only* way context ever changes — no other verb touches it as
a side effect of running, even one whose own `path` fully resolves to something that could
sensibly become the new context (`create_folder`, notably, used to leave its own path as the
new context; it no longer does — every verb only *reads* context, never writes it). `use`'s
own `path` must always be a whole, absolute path — unlike every other verb, it's never
resolved against whatever context already exists, and never accepts a leading `.`/`../`. It
does make a live check that `path` actually exists in the catalog first, raising if it
doesn't, rather than silently pointing context somewhere later calls would fail against
anyway; `use(None)`/`use("")` skip that check and reset context to `"catalog"` instead.
Every other verb's own `path`/`source_path`/
`target_path` — including `data_copy`/`data_move`/`create_view`'s two, each resolved
independently against that same context — still resolves against the current context once
one is set — with or without a leading `.` (`"2027"` and `".2027"` mean the same thing) —
unless it already starts with `catalog` (this deployment's one real root source), in which
case it's always taken literally as absolute rather than appended to the context, dot or
not. That's what makes pairing a relative `source_path` with a genuinely unrelated absolute
`target_path` in the same `data_copy`/`data_move`/`create_view` call unambiguous without
needing a dot on the absolute side: a real absolute path here always starts with `catalog`.
One or more leading `../` (or a bare `..`) instead walks up that many levels of the context
first, everywhere a relative path is understood except inside `use` itself —
`"../water_temperature"` is a sibling of the context, `"../../water_temperature"` a level
further up; `"."`/`"./"` alone (no name after either) mean the context itself. `get_context()`
shows what the context currently is — always exactly what `use`/`set_context` last stored —
and `use(path)` prints the same thing itself (`context set to <path>`), since there's nothing
to commit — `use` never queues anything, so `%catalog`'s usual auto-commit would otherwise
have nothing to show:

```python
%%catalog use("bwd.reference")
set_tags("water_temperature", ["reviewed"]) # bare, no dot — same as ".water_temperature"
create_folder(".2027")
create_folder(".2027") # creates bwd.reference.2027 for real, but leaves context untouched

%catalog use("bwd.reference.2027") # use() always takes the whole, absolute path
%catalog get_context() # -> 'bwd.reference' — still what use() set, not create_folder
%catalog use("bwd.reference.2027") # prints: context set to 'bwd.reference.2027'
%catalog get_context() # -> 'bwd.reference.2027'
%catalog list(".") # '.' alone -> the context itself: same as list("bwd.reference.2027")
%catalog set_tags("../water_temperature", ["archived"]) # ../ works for any verb except use()
%catalog get_tags("catalog.other_root.assessments") # starts with 'catalog' — absolute, not appended
%catalog data_copy(".water_temperature", "other_root.archive.water_temperature_2027")
# ^ bare, but data_copy/data_move/create_view always
# take a bare target_path/source_path literally
%catalog data_copy(".water_temperature", "catalog.other_root.archive.water_temperature_2027")
# ^ starts with 'catalog' — absolute, not appended either
%catalog data_copy(".water_temperature", "archive")
# ^ bare, no dot — relative too, appends to the context
```

`get_wiki(path)`, `get_tags(path)`, `list(path)` (every table/view under `path`, at any depth)
Expand Down
Loading
Loading