Telemetry by pblazej · Pull Request #1396 · livekit/rust-sdks

pblazej · 2026-09-03T11:26:43Z

Telemetry

Draft: livekit-telemetry, a shared client-telemetry core exposed over UniFFI, implementing the "Client Telemetry" design doc. The wire spec — events, attributes, cadence and upload policy — lives in livekit-telemetry/SPEC.md.

Design

One pipeline per process and one Session (trace id) per room: SDKs push warn/error logs, spans, getStats() readings and device state; the core owns everything downstream.
Readings are windowed and events batched on device, gzipped, written to a write-ahead cache (memory or file) and exported as OTLP/HTTP through a host-provided TelemetryTransport.
Telemetry never wins over media: a per-tick batch budget, uploads held while connecting or while WebRTC reports the encoder bandwidth-limited, and a cadence stretched by thermal, memory, battery and network state.
Self-telemetry (lk.telemetry.report), a flood guard, disk-full and collector-throttling handling are built in.

Integration

livekit-uniffi exposes Telemetry and TelemetrySession; the Swift SDK is the reference integration: livekit/client-sdk-swift#1108. The first commit unifies the workspace on prost 0.14 and is cherry-pickable on its own.

Size

iOS arm64 liblivekit_uniffi.dylib: +180 KiB over main, within the doc's ~300 KB budget (the iOS size gate is already stale for main).

Move livekit-protocol, livekit, livekit-api, livekit-data-stream and livekit-uniffi from a pinned prost 0.12 to the workspace prost 0.14, with pbjson/pbjson-types 0.6 -> 0.9. The committed prost-generated code compiles unchanged, so no regeneration is needed. Binaries combining these crates (livekit-ffi already used the workspace prost) now link a single prost, and crates that depend on prost 0.14 types (e.g. opentelemetry-proto) no longer drag in a second copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Shared client telemetry core, consumed by the Rust SDK and (via livekit-uniffi) by the platform SDKs. Mirrors the OpenTelemetry Logs SDK shape: - Telemetry::emit synchronous, never-blocking entry point - Store bounded in-memory queue, drop-oldest - Exporter actor: 1 s tick, batches of 512, OTLP encode, bounded retries honoring Retry-After, drop on reject, go silent when the collector disables telemetry - TelemetryTransport the only injection point: the core composes URL/headers/body, the transport moves bytes; NetTransport over livekit-net behind `net` - FileCache (storage_dir) one file per encoded batch, .tmp -> rename, oldest-first replay, drop-oldest above max_storage_bytes, 24 h max age; shutdown spills the queue to disk before trying the network; throttled/rejected/disabled data is never written OTLP types come from opentelemetry-proto (gen-tonic-messages only); its SDK dependencies are dead code and LTO removes them. SPEC.md seeds the event catalog (lk.ping). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Thin uniffi::Object wrapper around livekit_telemetry::Telemetry that spawns the exporter on the global tokio runtime. Hosts pass a TelemetryConfig record and implement the async TelemetryTransport foreign trait (e.g. a URLSession or OkHttp POST); TelemetryEvent/Attribute/Severity/ExportError cross the boundary as records and enums from the livekit-telemetry component. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Sends one `lk.ping` to an OTLP/HTTP collector (default: a local grafana/otel-lgtm on :4318) through NetTransport. LK_OTLP_ENDPOINT overrides the URL; LK_TELEMETRY_DIR enables the on-disk cache so the offline -> online replay can be exercised by stopping and starting the collector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…sport Persistence was write-behind: only batches that failed after retries were written, plus a spill on shutdown. That survives being offline or killed but not a crash, which loses up to a flush interval of events - the seconds a crash investigation wants. Every client SDK surveyed (Sentry envelopes, Datadog batch files, opentelemetry-android disk buffering) writes before sending, so the cache is now the queue between exporter and transport. - BatchCache trait: push(id, body) / pending / read / remove / clear. Ids are exporter-minted `<unix_ns>-<seq>-<events>`: sortable, age-bearing, countable. - MemoryCache (default): failed uploads wait for the next attempt instead of being dropped after three tries. FileCache: the former persist.rs, selected by `storage_dir`. Telemetry::with_cache accepts any other impl. - Exporter collapses to one path: enqueue (encode -> cache), then upload (oldest-first -> transport -> remove); shutdown only ignores the upload backoff. Retry-After keeps cached batches but drops new ones for its window; Disabled empties the cache. - max_storage_bytes -> max_cache_bytes (applies to both implementations). ios-arm64 UniFFI dylib: 1,193,368 B (+16.4 KB over write-behind), 13.7 KiB over SPM_SIZE_LIMIT_BYTES - the gate bump is left as a separate decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Self-telemetry (the Sentry "client report" shape, reason names from the OTel SDK self-metrics conventions): Counters shared by store and exporter track every way data is lost - queue_full, cache_error, rejected, throttled, disabled - plus uploads sent and failed attempts. Telemetry::stats() exposes them; whenever something went wrong since the previous report the exporter appends one lk.telemetry.report event to the next batch. Never an extra request, never persisted on its own, silent when nothing is wrong. Device state: the host pushes DeviceState { thermal, low_power_mode, app_state } through Telemetry::set_device_state. The core emits the lk.device.{thermal,low_power,app_state}.changed events (initial value on the first call) and multiplies the flush interval by cadence_factor(): serious thermal x2, critical x4, low power x2, background x2, capped at 4x; entering the background flushes once immediately. The OS APIs stay on the host - no Rust crate can read them without a JVM/ObjC bridge. UniFFI: Telemetry.stats(), Telemetry.setDeviceState(); TelemetryStats, DeviceState, ThermalState, AppState records/enums. ios-arm64 dylib +744 B. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… attributes The remaining v0 surface from the design doc: - Log records: a TelemetryEvent with an empty name is a plain OTLP log record (severity + body, no event_name). Only Warn/Error leave the device; emit() drops Trace/Debug/Info. - Flood guard: discrete events are capped at max_events_per_10min (300); the excess is dropped and counted as rate_limited (stats + report). RTC windows and self-telemetry are exempt. - RTC stats: platforms push raw getStats() readings as RtcStatsSample every 1-2 s; StatsWindows folds them per track and direction into one lk.rtc.stats.sample per stats_window_ms (15 s, stretched with the cadence): cumulative counters as the last value (monotonic, W3C webrtc-stats model), gauges as min/max/avg. Windows close on the stats tick, on background and on shutdown; the window ticker starts a full period out so early readings are not shipped as a zero-length window. - Session attributes: Telemetry::set_attribute(key, value) attaches lk.room.sid / lk.participant.identity / app correlation ids to every record at encode time without overriding explicit ones. UniFFI: Telemetry.recordStats(sample:), Telemetry.setAttribute(key:value:), RtcStatsSample / TrackKind / StreamDirection. ios-arm64 dylib 1,211,720 B (+17.6 KB, mostly the 21-field RtcStatsSample record lowering). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

livekit-runtime was removed upstream (#1375): livekit-telemetry now uses tokio::time directly. livekit-net gained its UniFFI bridge (#1290) and livekit-signaling/livekit-region were extracted from livekit-api (#1345); the new crates are moved onto the workspace prost like the rest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two ways for a host to move telemetry bytes, chosen per binding: - Telemetry(config, transport: Option<TelemetryTransport>): None falls back to the HTTP client the host registered with livekit-net (set_http_client), so a platform that already brought its client for signaling needs nothing more. Fails with TelemetryError::NoTransport when neither exists. - Telemetry::new_pulled(config, TelemetryExportQueue): Rust never calls into the host. The exporter queues each ExportRequest; the host awaits next(), performs it on its own thread and reports back with complete(id, error). The second exists for uniffi-dart: its foreign-trait callbacks are isolate-bound (Pointer.fromFunction) and the Dart VM aborts with "Cannot invoke native callback outside an isolate" when the exporter invokes send() from a tokio worker - reproduced with a Dart TelemetryTransport. Swift and Kotlin callbacks are thread-agnostic and keep using the trait directly. support/dart/test/telemetry_test.dart covers the pull path end to end (emit + stats window -> two ExportRequests served from Dart) and the no-transport error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A span is one attempt at an operation; the session is the trace. The core mints the trace id when the pipeline starts (rand, non-zero) and attaches it to every log record; records emitted inside a span carry its span_id too. - Spans registry: begin_span(name, kind, parent) -> u64 handle, add_span_event(handle, name, attrs) for checkpoints, end_span(handle, outcome, error_type, attrs). Handles are opaque u64s; ambient context is the platform's job. OTel span limits (128 events/attrs), 256 open spans, finished spans bounded like the event queue (drop-oldest, counted). - Outcome: OTel status only knows Unset/Ok/Error and instrumentation should not set Ok, so ok and cancelled export as Unset and every span carries lk.outcome = ok|error|cancelled (+ error.type and the status message on error). Checkpoints are span events in the span's own envelope (OTEP 4430 keeps that legal); real events stay log records pointing at the span. - Wire: opentelemetry-proto `trace` types, ExportTraceServiceRequest; a second signal through the same BatchCache - batch ids are now <unix_ns>-<seq>-<count>-<l|t> (old ids read as logs) and upload picks traces_endpoint, derived from the logs endpoint by replacing the last "logs" path segment (covers /v1/logs and /observability/logs/otlp/v0) or set explicitly. - UniFFI: traceId(), beginSpan, addSpanEvent, endSpan; SpanKind, SpanOutcome; TelemetryEvent.spanId. - SPEC.md: span rules, lk.connect and lk.reconnect definitions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Telemetry must never win over media, so uploads are shaped as well as batched: gzip bodies (level 1, compressed when cached), a `Priority: u=7` hint, at most `max_batches_per_upload` (4) cached batches per tick, and holds while `lk.connect`/`lk.reconnect` are open, while WebRTC reports an outbound track bandwidth-limited (derived from the stats we already get), or while the device asks for quiet (constrained network, battery <= 10 % unplugged). A hold lasts 60 s at most; shutdown drains without the budget. `DeviceState` grows memory pressure, network path (type, expensive, constrained) and battery; each feeds `cadence_factor` and three new `lk.device.*.changed` events (battery on bucket crossings only, OTel `hw.battery.*` names, `network.connection.type`). A CPU-limited encoder (`qualityLimitationDurations.cpu`) doubles the cadence too - CPU is never measured directly, per the design doc. The exporter schedules ticks as `next = last + period` so a cadence change applies to the pending tick in both directions, and reads the device state synchronously (a command-only copy raced with emit + flush). `session.id` joins the resource attributes. SPEC.md documents the events, the cadence table and the upload policy. `target-*/` is ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…gnostics The queue tracks its approximate size: crossing `flush_threshold_bytes` (256 KiB) wakes the exporter instead of waiting for the tick, and a request never carries more than `max_batch_bytes` (1 MiB before compression) - the design doc's "flush every 15 s or at 256 KB; single POST <= 1 MB". Both go through the same budget and holds, so an early flush is never a burst. `emit_custom(name, attributes)` is the stringly-typed escape hatch next to the `lk.*` catalogue: events ship as `custom.<name>`, so they can neither collide with nor spoof SDK events and the backend can quota the namespace. Self-diagnostics audit: uploads now report bytes on the wire and split timeouts from other failures; cache eviction (max size / age) is counted as `dropped.cache_full` instead of being silent (`BatchCache::push` returns the evicted ids); holds that hit the 60 s cap are counted; and one `lk.telemetry.report` is always emitted at shutdown as the session summary, so healthy sessions contribute denominators too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…d destination The design doc wants buffering from SDK init and flushing once a granted token is available; the Swift integration started the pipeline at connect(). The core now models what that needs: - `Telemetry` is the process pipeline (queue, cache, exporter, device state, self-telemetry). `Telemetry::begin_session()` returns a `Session` - one room, one call - with its own trace id and attributes; queued records, spans and RTC windows are filed under the session that produced them and carry `session.id` as a record attribute (no longer a resource attribute). Everything emitted outside a session belongs to the pipeline's process session. A log record emitted with a span id is filed under that span's session, so SDK loggers need no session plumbing. - `TelemetryConfig.endpoint` is optional. Without one the pipeline buffers and caches; uploads wait (uncapped) for `set_destination(endpoint, headers)`, which the first connect supplies from the server URL and token. Calling it again redirects later batches. UniFFI gains `TelemetrySession`, `begin_session` and `set_destination`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

github-actions · 2026-09-03T11:27:07Z

Changeset incomplete

This PR's changeset is missing version bumps for packages that are affected by the change. The following packages still require a bump:

livekit-common
livekit-datatrack
livekit-ffi
livekit-signaling
livekit-token

Already covered:

livekit (patch)
livekit-api (patch)
livekit-data-stream (patch)
livekit-protocol (patch)
livekit-telemetry (minor)
livekit-uniffi (minor)

A package must be bumped when its own files change, and whenever a package it depends on is bumped (so downstream consumers get a matching release).

Click here to create a changeset for the missing packages

The link pre-populates a changeset file with patch bumps for the missing packages. You can also add them to your existing changeset. Edit the bump types as needed before committing.

If this change doesn't require a version bump, add the internal label to this PR.

`Telemetry::set_attribute` now attaches an attribute to every record of every session (an `enduser.id`, a tenant), merged at export after the session's own attributes and never overriding a record's explicit key. Session identity stays on `Session::set_attribute`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Log viewers key their line on the body and not every backend surfaces event_name yet, so attribute-only events (RTC windows, device changes) rendered as empty lines in Loki. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…severity Events left the device with attributes only; OTel calls an event's body its display message, log viewers key their line on it, and Loki has no EventName mapping yet, so RTC windows and device changes rendered as blank info lines. Every event now carries a one-line summary - `video outbound: 1204 kbps, loss 0.4%, rtt 48 ms, 30 fps`, `battery: 95%, charging`, `telemetry: 7 batches sent (4223 B), 0 failed, 0 dropped, 0 cached` - and `otel.event.name` (semconv 1.39) so backends without EventName can still tell events from logs. The name remains the last-resort body. `TelemetryConfig.log_severity` (default warn) is the threshold a plain log record needs to leave the device; events are not subject to it. Severity is now ordered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A record naming a span is filed under that span's session, but the lookup only knew open spans. SDK log paths hop threads, so a warning logged right before its span ends (a failing publish) arrived after the span had moved on and fell into the process session. Spans now remember their session for the last 1024 spans, open, ended or exported. Found by the e2e test that checks span-to-log attribution. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Holds, the hold cap, pauses after failures, the missing destination and every sent batch now log at debug, so a host's console shows why nothing is leaving the device. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

WebRTC reports qualityLimitationReason=bandwidth for minutes during a normal ramp-up and for as long as an encoder stalls. On a real iPhone a camera track at 0 kbps kept the hold alive for 8 minutes; only the 60 s cap let one batch per minute out while the cache grew by 11 per minute. Yielding to media is the transport's job (Priority: u=7, background service class). The cpu counter still stretches the cadence. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>