Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ consistent pattern (parameter class → request → result object), across all f
|---|---|
| Ingestion | Registering providers, ingesting data (unary, streaming, bidirectional), checking request status, subscribing to live data |
| Query | Retrieving time-series data as samples, buckets, or tables; retrieving PV, provider, and ingestion statistics |
| Annotation | User-defined PV metadata, machine configuration, datasets, annotations, and export |
| Annotation | User-defined PV metadata, machine configuration, per-sample status, datasets, annotations, and export |
| Ingestion Stream | Event subscriptions that fire when a data condition is triggered |

**Higher-level application features.** An `MldpApplication` layer built on top of the API
Expand Down Expand Up @@ -63,6 +63,13 @@ for their service.
active at a given instant. Covers save/get/query/iterate/delete for both configurations and
activations, plus `get_active_configurations()`, with the `ConfigurationQuery` and
`ConfigurationActivationQuery` helpers.
- **Sample status API** — `client.annotation.sample_status`. Assign a status code to individual PV
samples at exact instants — a quality flag, a model's anomaly score, an operator override — and
query time-series data with flagged samples filtered out. Covers `save_sample_statuses()`,
`query_sample_statuses()`, `iter_sample_statuses()`, `iter_sample_statuses_stream()`, and
`delete_sample_statuses()`, with the `SampleStatusFrame` / `SampleStatusColumn` builders and the
`SampleStatusFilter` query selector. This is the replacement for `DataValue.ValueStatus`, which
was removed in 1.16.0.
- **v2 query API (samples)** — `client.query`. Sample-oriented time-series retrieval over a
half-open time range, selecting PVs by name list, name pattern, or metadata query, and
optionally restricting to intervals where a machine configuration was active. Unary with
Expand All @@ -89,8 +96,9 @@ pydantic-settings), TLS-capable channel creation, hierarchical logging, three-ti
(gRPC errors, business-logic errors, unexpected exceptions), comprehensive type hints, and a unit
and integration test suite.

Note the v2 query API comes from unreleased dp-grpc work and will not work against a
`rel-1.14.0` server.
Note this package's version tracks the dp-grpc version its stubs were generated from, so a server
older than your `dp_python_lib` will not implement everything listed here. The v2 query API needs a
`rel-1.15.0` or later server; the sample status API needs `rel-1.16.0` or later.

## TODO

Expand Down Expand Up @@ -120,7 +128,9 @@ Note the v2 query API comes from unreleased dp-grpc work and will not work again

**Project infrastructure**

- CI workflow(s) for running regression tests and publishing release artifacts
- Publishing to PyPI. The release workflow has the job wired up but disabled; everything else —
unit tests across Python 3.10-3.13, lint and format checks, the cookbook snippet checker, and
signed release artifacts — runs in CI today.

## Installation

Expand All @@ -137,6 +147,11 @@ pip install -e .[analysis]
pip install -e .[dev]
```

**Upgrading from 1.15.0 or earlier:** 1.16.0 raises the `grpcio` floor to 1.84.0, because the
regenerated stubs require it. `pip install` picks that up, but an existing editable install will
not upgrade it on its own — the stubs then fail at import with a version mismatch naming the
required release. `pip install -e . --upgrade` resolves it.

Point the client at your MLDP services with an `mldp-config.yaml` file or `MLDP_*` environment
variables — see [Creating and connecting a client](doc/cookbook/connecting.md).

Expand Down Expand Up @@ -182,6 +197,8 @@ the recipes share one continuous worked example drawn from an accelerator facili
| [Cataloguing PVs](doc/cookbook/pv-metadata.md) | Recording what a PV is, then finding PVs by property instead of by name |
| [Recording machine configuration](doc/cookbook/machine-configuration.md) | Defining configurations, recording when each was active, and answering "what was the machine doing at 18:04?" |
| [Querying time-series data](doc/cookbook/query.md) | Retrieving samples by PV, metadata, or machine configuration, and converting to pandas / NumPy / Excel |
| [Labeling samples](doc/cookbook/sample-status.md) | Recording per-sample status codes, reading them back, and querying data with flagged samples excluded |
| [DataSets and annotations](doc/cookbook/datasets-and-annotations.md) | Naming a region of the archive, attaching analysis results with column-level provenance, and exporting |

Every Python snippet in the cookbook is mechanically syntax- and type-checked against the
installed package.
Expand Down
8 changes: 5 additions & 3 deletions doc/cookbook/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,11 @@ Attribute names and values are the facility's; tag values are illustrative place

## Conventions used in recipes

- Each recipe states the release it was **verified against**. Most of the API is stable since
dp-grpc `rel-1.14.0`, the current release. The [v2 query API](query.md) is the exception: it
comes from unreleased dp-grpc work and **will not work against a `rel-1.14.0` server**.
- **Recipes note the release an API arrived in, where it matters.** This package's version tracks
the dp-grpc version its stubs were generated from, so a server older than your `dp_python_lib`
will not implement everything documented here. Where a recipe uses something added in a
particular release, it says so in the body — the [v2 query API](query.md) needs a `rel-1.15.0` or
later server, and [sample status](sample-status.md) needs `rel-1.16.0` or later.
- Snippets omit imports and client construction except where a recipe is specifically about those
things. Each recipe lists the imports its examples assume.
- Examples check `result_status.is_error` before reading a payload. This is not ceremony: the
Expand Down
39 changes: 25 additions & 14 deletions doc/release-notes/rel-1.16.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ int32 status code to **one PV sample at one instant** — an ML model labeling s
rule engine flagging out-of-range values, an operator marking a handful of suspect points. It is the
designated replacement for `DataValue.ValueStatus`, removed in this same release.

Reference: [CLAUDE.md, Sample Status API](../../CLAUDE.md#sample-status-api-annotation-service).
Worked examples: [Sample status cookbook](../cookbook/sample-status.md).
Reference: [CLAUDE.md, Sample Status API][claude-sample-status].
Worked examples: [Sample status cookbook][cookbook-sample-status].

### Methods

Expand Down Expand Up @@ -131,9 +131,9 @@ Three feature clients on the `annotation` facade: `client.annotation.datasets` (
A DataSet names a region of the archive; an Annotation describes one or more DataSets and may own a
Calculations payload of derived values; export writes any of it to a file on the server.

Reference: [CLAUDE.md](../../CLAUDE.md#datasets-annotations-and-export-api-annotation-service).
Worked example: [Data sets and annotations cookbook](../cookbook/datasets-and-annotations.md).
Design record: [`plan/tickets/6/plan.md`](../../plan/tickets/6/plan.md).
Reference: [CLAUDE.md][claude-datasets].
Worked example: [Data sets and annotations cookbook][cookbook-datasets].
Design record: [`plan/tickets/6/plan.md`][plan-6].

With this, `AnnotationClient` covers **every implemented `DpAnnotationService` feature area** —
`.pv_metadata`, `.machine_config`, `.sample_status`, `.datasets`, `.annotations`, `.export` — all
Expand Down Expand Up @@ -295,7 +295,7 @@ the #6 helpers; this back-ports it to all seven — `PvMetadataQuery`, `Configur
The **key** check is load-bearing on the two v2 query selectors, where the server does not validate it:
a blank key would reach Mongo as an existence test on `"attributes."` and silently match nothing.

Design record: [`plan/tickets/40/plan.md`](../../plan/tickets/40/plan.md).
Design record: [`plan/tickets/40/plan.md`][plan-40].

## Browse-all queries (Issue #41)

Expand All @@ -312,7 +312,7 @@ and a token. That is why `iter_*` is the right call for browsing.
The v2 query methods are deliberately **not** included: `QueryParams` still requires a PV selector or
config criteria, since a time-series query with no selection is unbounded rather than a browse-all.

Design record: [`plan/tickets/41/plan.md`](../../plan/tickets/41/plan.md).
Design record: [`plan/tickets/41/plan.md`][plan-41].

Both #40 and #41 are covered by
`tests/integration/test_query_helper_relaxations_integration.py`, which is live-server work a unit test
Expand All @@ -337,7 +337,7 @@ streamed message — error results included, for the public `iter_*` wrapper to
`RuntimeError` — which is a different contract from returning a single result.

`_dispatch` is itself unit-tested (`tests/unit/test_service_api_client_base.py`), which the duplicated
blocks never were. Design record: [`plan/tickets/14/plan.md`](../../plan/tickets/14/plan.md).
blocks never were. Design record: [`plan/tickets/14/plan.md`][plan-14].

## Dependency floors

Expand All @@ -354,11 +354,10 @@ is easy to miss until it reaches a pinned environment.

## Documentation and process

Two new cookbook recipes: **[Sample status](../cookbook/sample-status.md)** and
**[Data sets, annotations, export](../cookbook/datasets-and-annotations.md)**.
**[API conventions](../cookbook/conventions.md)** documents the browse-all form and the key-only
attribute exception. The PV metadata, machine configuration, and query recipes are updated for the
relaxed helpers.
Two new cookbook recipes: **[Sample status][cookbook-sample-status]** and **[Data sets,
annotations, export][cookbook-datasets]**. **[API conventions][cookbook-conventions]** documents
the browse-all form and the key-only attribute exception. The PV metadata, machine configuration,
and query recipes are updated for the relaxed helpers.

Every cookbook snippet is now checked by `.dev/tools/check-cookbook-snippets.py`, which type-checks
each example against the installed package with mypy and runs in CI. It catches wrong attribute and
Expand All @@ -367,7 +366,8 @@ their interpreter.

**Plan documents are now version-controlled** under `plan/tickets/<issue>/`, the convention dp-grpc and
dp-service already use, replacing the gitignored `.dev/plan/issue-<n>/`. Plans there were invisible to
reviewers, to CI, and to anyone working from a fresh clone. See [`plan/README.md`](../../plan/README.md).
reviewers, to CI, and to anyone working from a fresh clone. See
[`plan/README.md`][plan-readme].

Per-recipe "Verified against" headers are gone, matching dp-grpc #141: the convention asserts when
someone last checked a recipe, so it decays at every version bump — five of the seven recipes were
Expand Down Expand Up @@ -396,3 +396,14 @@ could not parse its own config and failed CI on every sync PR. The step was rem
(dp-grpc #153): this repo derives its version from its own git tag via setuptools-scm, so there was never
a literal version for it to bump, and ruff's key was the only line the pattern could reach. The step had
in fact been a no-op since before `rel-1.15.0`.

[claude-sample-status]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/CLAUDE.md#sample-status-api-annotation-service
[claude-datasets]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/CLAUDE.md#datasets-annotations-and-export-api-annotation-service
[cookbook-sample-status]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/doc/cookbook/sample-status.md
[cookbook-datasets]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/doc/cookbook/datasets-and-annotations.md
[cookbook-conventions]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/doc/cookbook/conventions.md
[plan-6]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/plan/tickets/6/plan.md
[plan-14]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/plan/tickets/14/plan.md
[plan-40]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/plan/tickets/40/plan.md
[plan-41]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/plan/tickets/41/plan.md
[plan-readme]: https://github.com/osprey-dcs/dp-python-lib/blob/rel-1.16.0/plan/README.md