Harden telemetry delivery and refine recipe metadata

[bmehta001]

Describe your changes

This is a clean-history replacement for #2443. It supersedes that pull request for review purposes but intentionally does not modify or close #2443 yet. The single replacement commit preserves the finalized reviewed tree exactly (8403987295683c7eaf182777cb859858835abe18) while removing the accumulated development history.

The preserved implementation:

• makes telemetry opt-out complete across CLI, Python API, Docker, and workflow entry points;
• delivers Heartbeat and recipe events durably through a retained SQLite queue and bounded uploader drain;
• retains queued rows safely across transient failures, shutdown, and process contention;
• aligns ORT identity and CI behavior while keeping CI recipe delivery bounded;
• enforces the telemetry privacy boundary with recursive metadata, traceback, path, URL, and credential redaction;
• covers all Olive commands, init, API, Docker, and workflow paths with focused tests.

For historical review context, see Hitesh's prior CHANGES_REQUESTED review commit 7427d751acf1293c28beba76b20d5df99fdfc36c.

Validation:

• Windows Python 3.13 targeted telemetry/init/API/CLI/Docker/workflow matrix: 267 passed, 1 skipped; the sole excluded integration failure (test_capture_onnx_graph_integration) reproduces identically on pristine upstream/main in the current Transformers environment.
• WSL/Linux telemetry: 153 passed, 1 skipped.
• Ruff check and Ruff format check: clean.
• Worktree and feature diff checks: clean.
• Import smoke: replacement 50.3s vs pristine main 55.4s in the same environment.
• Fresh material code review: No material findings.

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass. (One environment-dependent API integration baseline failure is documented above and reproduced on main.)
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

Release-note summary: Olive telemetry now honors opt-out consistently and improves privacy-safe, durable delivery of Heartbeat and recipe metadata.

(Optional) Issue link

Supersedes #2443 with clean history; the original PR remains unchanged for now.

[Copilot]

Pull request overview

This PR replaces Olive’s telemetry pipeline with a standard-library implementation that persists events to a per-user SQLite queue and uploads them in the background, while also refining “recipe” metadata emission and making telemetry opt-out consistent across entry points (CLI, Python API, Docker, workflows).

Changes:

• Replaced the OpenTelemetry-based logger/exporter with a durable SQLite offline store + background uploader, plus stricter privacy redaction for strings/config snapshots.
• Added/updated recipe-telemetry capture at workflow/CLI/Docker boundaries (including CI “recipe-only” behavior and forwarding opt-out/CI signals into containers).
• Expanded tests and updated documentation; removed opentelemetry-sdk from runtime requirements.

Reviewed changes

Copilot reviewed 39 out of 40 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/workflows/test_workflow_run.py	Adds focused tests asserting recipe-result telemetry metadata, redaction behavior, hashing stability, and Docker-parent recipe semantics.
test/systems/docker/test_docker_system.py	Adds tests for CI/env forwarding and for disabling inner-container recipe/error telemetry in the workflow runner.
test/conftest.py	Ensures tests apply telemetry opt-out before singleton construction and shuts down telemetry at session end.
test/cli/test_cli.py	Adds coverage for CLI opt-out latching, telemetry lifecycle, and workflow-run recipe metadata injection.
test/cli/test_api.py	Verifies Python API surfaces latch disable_telemetry behavior.
test/cli/init/test_init_command.py	Adds init command telemetry action/error emission tests.
requirements.txt	Removes opentelemetry-sdk dependency from runtime requirements.
README.md	Updates telemetry blurb wording and formatting.
olive/workflows/run/run.py	Emits recipe-result and error telemetry around workflow execution, with CI-bounded shutdown behavior.
olive/telemetry/utils.py	Hardens telemetry base-dir resolution to require absolute per-user directories.
olive/telemetry/uploader.py	Introduces background uploader to drain SQLite queue with retry/split/ack/delete behavior and single-drainer locking.
olive/telemetry/telemetry.py	Reworks telemetry singleton to persist events to SQLite and upload via background thread; adds CI detection + full opt-out latching.
olive/telemetry/telemetry_redaction.py	Adds recursive redaction for secrets/paths/URLs/free-text and caps payload size.
olive/telemetry/telemetry_extensions.py	Updates action/error/recipe helpers, adds exception de-duplication marker, and formats tracebacks without source lines.
olive/telemetry/recipe_telemetry.py	Adds recipe metadata extraction, hashing with redaction, model/source classification, and config/package override snapshot sanitization.
olive/telemetry/process_lock.py	Adds cross-platform advisory single-holder lock for multi-process uploader coordination.
olive/telemetry/offline_store.py	Adds SQLite-backed durable FIFO event store with WAL/busy-timeout and permission hardening.
olive/telemetry/library/transport.py	Switches HTTP transport to urllib with worker-thread timeout bounding and retryability classification.
olive/telemetry/library/telemetry_logger.py	Deletes OpenTelemetry logger facade.
olive/telemetry/library/serialization.py	Tightens serialization (finite floats, deterministic ordering, key scrubbing, allow_nan=False).
olive/telemetry/library/retry.py	Deletes OpenTelemetry-era retry helper (superseded by uploader/store semantics).
olive/telemetry/library/options.py	Removes requests/http factory dependency and adjusts exporter options.
olive/telemetry/library/exporter.py	Deletes OpenTelemetry exporter implementation.
olive/telemetry/library/event_source.py	Deletes OpenTelemetry diagnostic event-source layer.
olive/telemetry/library/callback_manager.py	Deletes callback manager used by OpenTelemetry exporter path.
olive/telemetry/library/init.py	Updates package exports/docs to reflect stdlib-only building blocks.
olive/telemetry/deviceid/deviceid.py	Reworks device-id generation/publication, adds locking, and switches to hashed device-id reporting.
olive/telemetry/deviceid/_store.py	Hardens device-id storage (atomic write, permissions, size limits, registry type checks).
olive/telemetry/deviceid/init.py	Updates export to hashed-device-id function.
olive/telemetry/constants.py	Removes embedded connection-string constant file.
olive/telemetry/init.py	Exposes disable_telemetry alongside Telemetry and action.
olive/systems/docker/workflow_runner.py	Disables inner telemetry in container workflow runner and performs bounded shutdown flush.
olive/systems/docker/docker_system.py	Forwards CI + full telemetry opt-out into container env.
olive/cli/run.py	Captures explicit CLI overrides into recipe telemetry metadata and disables error telemetry for this path.
olive/cli/launcher.py	Latches opt-out before telemetry construction and ensures shutdown on all exit paths.
olive/cli/init/init.py	Adds telemetry options to init and wraps run with action telemetry.
olive/cli/base.py	Adds generated CLI recipe telemetry metadata to workflow execution path.
olive/cli/api.py	Latches opt-out when disable_telemetry is passed via Python API.
docs/Privacy.md	Updates privacy/telemetry documentation for opt-out semantics, CI behavior, SQLite durability, and redaction guarantees.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 39 out of 40 changed files in this pull request and generated no new comments.

Suppressed comments (1)

olive/telemetry/uploader.py:223

• When an individual offline-store row is too large to fit into an empty PayloadBuilder (builder.is_empty), drain_once drops the row via _finish_handled_rows but the returned DrainResult reports it as delivered=1. This makes the delivered counter inaccurate for oversized/poison rows (they're deleted, not delivered), which can mislead any future accounting or diagnostics that use DrainResult.delivered.

                if builder.is_empty:
                    return self._finish_handled_rows([row_id], deadline)
                break

[Copilot]

Pull request overview

Copilot reviewed 39 out of 40 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 39 out of 40 changed files in this pull request and generated no new comments.

Suppressed comments (1)

olive/telemetry/telemetry_extensions.py:61

• log_recipe_result isn’t wrapped in a suppress(Exception) guard like log_action/log_error. If telemetry initialization ever raises unexpectedly, this helper could propagate an exception into production call sites. Please make it best-effort/exception-safe for consistency with the rest of the telemetry helpers.

def log_recipe_result(
    recipe_name: str,
    success: bool,
    metadata: Optional[dict[str, Any]] = None,
) -> None:

[Copilot]

Pull request overview

Copilot reviewed 39 out of 40 changed files in this pull request and generated no new comments.