Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 51 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"
```

# staging's latest release (early access) — pin to the tag the "staging" badge above shows
pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.15-staging"
pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.17-staging"
```

## Usage
Expand Down Expand Up @@ -189,7 +189,7 @@ examples. Install the extra this needs once: `pip install "EEADataLakehouse[note
import eea_datalakehouse.notebook # registers %catalog/%ingest — no %load_ext needed

%catalog data_copy("draft.raw_2026", "bwd.reference.water_temperature")
%catalog set_tags(".water_temperature", ["reviewed"])
%catalog set_tags("bwd.reference.water_temperature", ["reviewed"])

%ingest ingest(folder="./bw_2026", target_catalog_path="bwd.reference",
data_format="parquet", table_name="water_temperature")
Expand All @@ -201,27 +201,56 @@ parameters, description — rather than a raw Python signature, since its audien
custodian, not necessarily a developer. `%ingest help` does the same for `IngestSession`'s
(fewer) methods — handy when you don't remember an exact parameter name mid-notebook.

A freshly created session already has a context — it starts at `"catalog"` (the catalog
root), not empty — so a relative path works even before the first `use()` call.
`use(None)`/`use("")` reset it back to `"catalog"` the same way; `set_context(None)` (the
lower-level primitive `use` wraps, not normally called directly) is the one way left to
clear it to no context at all.

`%%catalog` (the cell-magic form) sets the context once, with `use(path)` on its magic line,
then runs every other line of the cell in order under that context, without repeating the full
path on each line. A leading `.` resolves any `path`/`source_path`/`target_path` against the
context (so `data_copy`/`set_tags`/`create_folder`/... all understand it); `use` alone also accepts a bare
path with no dot, once a context exists — `"2027"` and `".2027"` narrow it the same way there.
path on each line. `use` is the *only* way context ever changes — no other verb touches it as
a side effect of running, even one whose own `path` fully resolves to something that could
sensibly become the new context (`create_folder`, notably, used to leave its own path as the
new context; it no longer does — every verb only *reads* context, never writes it). `use`'s
own `path` must always be a whole, absolute path — unlike every other verb, it's never
resolved against whatever context already exists, and never accepts a leading `.`/`../`. It
does make a live check that `path` actually exists in the catalog first, raising if it
doesn't, rather than silently pointing context somewhere later calls would fail against
anyway; `use(None)`/`use("")` skip that check and reset context to `"catalog"` instead.
Every other verb's own `path`/`source_path`/
`target_path` — including `data_copy`/`data_move`/`create_view`'s two, each resolved
independently against that same context — still resolves against the current context once
one is set — with or without a leading `.` (`"2027"` and `".2027"` mean the same thing) —
unless it already starts with `catalog` (this deployment's one real root source), in which
case it's always taken literally as absolute rather than appended to the context, dot or
not. That's what makes pairing a relative `source_path` with a genuinely unrelated absolute
`target_path` in the same `data_copy`/`data_move`/`create_view` call unambiguous without
needing a dot on the absolute side: a real absolute path here always starts with `catalog`.
One or more leading `../` (or a bare `..`) instead walks up that many levels of the context
first, everywhere a relative path is understood, not just in `use` — `"../water_temperature"` is
a sibling of the context, `"../../water_temperature"` a level further up. `use` also makes a live
check that the resolved path actually exists in the catalog, raising if it doesn't, rather than
silently pointing context somewhere later calls would fail against anyway; `use(None)` clears the
context. `get_context()` shows what it currently is — always the full resolved path:
first, everywhere a relative path is understood except inside `use` itself —
`"../water_temperature"` is a sibling of the context, `"../../water_temperature"` a level
further up; `"."`/`"./"` alone (no name after either) mean the context itself. `get_context()`
shows what the context currently is — always exactly what `use`/`set_context` last stored —
and `use(path)` prints the same thing itself (`context set to <path>`), since there's nothing
to commit — `use` never queues anything, so `%catalog`'s usual auto-commit would otherwise
have nothing to show:

```python
%%catalog use("bwd.reference")
set_tags(".water_temperature", ["reviewed"])
create_folder(".2027") # create_folder still needs the dot — only use() makes it optional

%catalog use("bwd.reference") # back to a path that already exists
%catalog use("2027") # bare, no dot — same as use(".2027"); already created above
%catalog get_context() # -> 'bwd.reference.2027'
%catalog set_tags("../water_temperature", ["archived"]) # ../ works for any verb, not just use()
set_tags("water_temperature", ["reviewed"]) # bare, no dot — same as ".water_temperature"
create_folder(".2027") # creates bwd.reference.2027 for real, but leaves context untouched

%catalog get_context() # -> 'bwd.reference' — still what use() set, not create_folder
%catalog use("bwd.reference.2027") # prints: context set to 'bwd.reference.2027'
%catalog get_context() # -> 'bwd.reference.2027'
%catalog list(".") # '.' alone -> the context itself: same as list("bwd.reference.2027")
%catalog set_tags("../water_temperature", ["archived"]) # ../ works for any verb except use()
%catalog get_tags("catalog.other_root.assessments") # starts with 'catalog' — absolute, not appended
%catalog data_copy(".water_temperature", "catalog.other_root.archive.water_temperature_2027")
# ^ starts with 'catalog' — absolute, not appended either
%catalog data_copy(".water_temperature", "archive")
# ^ bare, no dot — relative too, appends to the context
```

`get_wiki(path)`, `get_tags(path)`, `list(path)` (every table/view under `path`, at any depth)
Expand All @@ -230,12 +259,15 @@ to commit or undo. `delete_view(path)`/`delete_table(path)` queue like every oth
unlike the idempotent `DROP ... IF EXISTS` `Catalog.deleteview`/`deletetable` wrap — require
`path` to already exist, and (like `delete_folder`) can never be undone. Every one of these
raises `CatalogOperationError` if `path` doesn't exist; `schema` also requires it to be a table
or view, not a folder:
or view, not a folder. `list`'s `path` may also be omitted (or `""`) to list the current
context itself — raises `CatalogSessionError` if none is set yet:

```python
%catalog get_wiki("bwd.reference.water_temperature")
%catalog get_tags("bwd.reference.water_temperature")
%catalog list("bwd.reference") # -> ['bwd.reference.water_temperature', ...]
%catalog use("bwd.reference")
%catalog list() # same result — path omitted, lists the context itself
%catalog schema("bwd.reference.water_temperature") # -> TableInfo(schema={...}, row_count=...)
%catalog delete_view("bwd.reference.old_view")
```
Expand Down Expand Up @@ -312,7 +344,7 @@ To pin to one specific release instead, use the exact tag the live badges under
%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.12"

# staging's latest release (early access)
%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.15-staging"
%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.17-staging"
```

Use the `%pip` magic rather than `!pip` — it installs into the kernel the
Expand Down
Loading