interface to modernized annotation API

[craigmcchesney]

The original MLDP gRPC definition includes an Annotation API for identifying and describing datasets in the archive, and uploading user calculations derived from those datasets. That API was developed early in the project and has now been modernized to the conventions of the newer annotation-service APIs (PV metadata, machine configuration, sample status): a complete CRUD method set, criteria-based paged queries, flat save requests, audit fields, typed calculations columns, column-level provenance, and ad-hoc export. This ticket adds the Python interface to the modernized API.

Epic: #10

Upstream API change: dp-grpc#132, merged as dp-grpc PR #145 (design record: dp-grpc/plan/tickets/132/plan.md, D1–D16). Same release: dp-grpc#143 (ValueStatus removal) and dp-grpc#245 (empty criteria = match all).

Server implementation: dp-service#248, landed in PRs #256, #261, #263, #264 (design record: dp-service/plan/tickets/248/plan.md). Not yet in a release: the latest tag everywhere is rel-1.15.0; this work targets 1.16.0.

Plan: plan/tickets/6/plan.md — the working document, with the verified message shapes, the server behaviors the client must encode, the design decisions and their rejected alternatives, the per-file task list, and the open questions. This ticket summarizes; where the two disagree, the plan wins, and where the plan and the protos disagree, the protos win. (The first plan under this repo's new plan/tickets/<N>/ convention, landing in #42; the link resolves once that merges.)

Scope

Three feature clients on the existing client.annotation facade, sharing the annotation channel like pv_metadata / machine_config / sample_status do:

RPC	Wrapper	Client
saveDataSet / getDataSet / queryDataSets / deleteDataSet	save_dataset / get_dataset / query_datasets + iter_datasets / delete_dataset, plus get_datasets(ids) batch fetch	client.annotation.datasets
saveAnnotation / getAnnotation / queryAnnotations / deleteAnnotation / getCalculations	save_annotation / get_annotation / query_annotations + iter_annotations / delete_annotation / get_calculations	client.annotation.annotations
exportData	export_data (dataset id and/or inline data blocks and/or calculations spec → HDF5 / CSV / XLSX)	client.annotation.export
patchDataSet / patchAnnotation	not wrapped — reserved "not yet implemented" placeholders, per the other patch* stubs	—

Alongside the wrappers, following the #5 / #9 / #8 pattern:

• Criterion helpers DataSetQuery (ids / owners / name / text / pv_names / tags / attributes) and AnnotationQuery (ids / owners / datasets / annotations / name / text / tags / attributes), with key-only attribute search and an optional criteria (the server now treats an empty list as match-all).
• Params and result classes for every method; data_block() and calculations_spec() builders; an ExportFormat enum that makes the rejected EXPORT_FORMAT_UNSPECIFIED unreachable.
• Calculations construction — a shared common.DataFrame builder module (data_frame.py: time axis, typed scalar columns, the legacy DataColumn escape hatch for sparse values, ColumnMetadata / ColumnProvenance.derivedFrom helpers, client-side count-match and uniqueness validation) that Add ingestion API client (full surface: ingestData + streaming) #17 (ingestion) extends rather than duplicates, since ingestionDataFrame is the same message.
• Reading calculations back — typed columns → Python (data_frame_conversions.py, shared with the interface to v2 bucket-oriented query API (queryBuckets / queryBucketsStream) #16 bucket query), and, behind the existing [analysis] extra, calculations_to_dataframes() / calculations_from_dataframes() pandas bridges.
• Unit tests per client and module; an integration test against a post-#264 annotation service (probing for the modernized API first, since a 1.15.0 server answers UNIMPLEMENTED); a datasets-and-annotations.md cookbook recipe; README / CLAUDE.md / conventions updates.

Prerequisite: stub sync

The committed stubs predate the modernization (last sync 2026-08-22; 30 RPCs, none of the seven new ones). Regenerating from dp-grpc main adds the seven RPCs, changes only annotation_pb2.py, annotation_pb2_grpc.py, and common_pb2.py in substance, and leaves all existing unit tests passing, so the sync merges on its own ahead of this work: #39 (dp-grpc run 34382676119). The run's newer grpcio-tools (1.83.1) also re-stamped every *_pb2_grpc.py banner, which raises the stubs' import-time grpcio requirement, so #39 carries the matching pyproject.toml floor bump — the same drift the #8 sync hit. The ValueStatus removal reaches this repo only as prose (four "deprecated / ignored" mentions to reword).

Phases

0. Stub sync (Sync Python gRPC stubs from dp-grpc #39); the Extract shared _dispatch helper for annotation service _send_* methods #14 _dispatch refactor first, as its own PR; the planning-convention docs PR (docs: add the issue #6 plan and adopt the plan/tickets convention #42, which also carries the PvMetadataClient query ignores limit=0 (truthiness bug) #13 regression test).
1. Low-level wrappers, criterion helpers, params / results, facade wiring, unit tests.
2. data_frame.py builders and the pure-Python read side, unit tests.
3. pandas bridges under [analysis], unit tests (skip-clean without the extra).
4. Integration test, cookbook recipe, README / CLAUDE.md / conventions.

Out of scope

patch* placeholders; array / image / struct / serialized column builders (#17); a download path for exports (no RPC exists — file_path is server-side); cascade delete (the server's two-step is the honest shape); the two follow-ups surfaced during triage — key-only attributes() on the five existing criterion helpers (#40) and optional criteria on the six existing query/iter methods (#41); bucket-query conversions (#16). #13 was found already fixed and closed.

Open questions

All eleven (client layout and naming, #14 first, calculations construction and pandas scope, optional criteria, validation depth, stub sync mechanism, PR structure, planning convention, integration-test server) were resolved 2026-09-09 as recommended; the plan records each question with its context and resolution. Implementation order: #39 → #14 → #6 in two PRs (wrappers and builders; pandas bridges, recipe, docs).