Metrics update

[DaemonLoki]

Why

• metrics were missing a few key things (e.g. realtime stats, time-to-first-answer from LLMs, realtime models, VLMs)
• we only had averages, no info about the number of data points
• no final snapshot (so when evaluating after a call, each consumer would have had to keep track of each emitted metrics event)
• advantages
  • better debugging (more metrics + see all metrics at the end of a call)
  • good tracking across all settings (STT -> LLM -> TTS, realtime, VLM)
  • don't need Prometheus setup for solid metrics support

Changes

• we have richer session metrics now
  • for every average we have a timestamp
  • turn count + total errors encountered
  • TTS: total latency + time-to-first-audio (TTFA)
  • Realtime: TTFA, session duration, response counts
  • VLM: TTFT( time-to-first-token)
• at the end of a call we deliver a call_metrics_summary event with all data (not changing the existing agent_metrics broadcast event)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2a29c80b-e379-4168-af86-49058f04c81c

📥 Commits

Reviewing files that changed from the base of the PR and between e9c0cb5 and a1c043c.

📒 Files selected for processing (6)

• CHANGELOG.md
• agents-core/vision_agents/core/agents/agents.py
• agents-core/vision_agents/core/observability/agent.py
• plugins/huggingface/vision_agents/plugins/huggingface/transformers_vlm.py
• plugins/nvidia/vision_agents/plugins/nvidia/nvidia_vlm.py
• tests/test_observability.py

🚧 Files skipped from review as they are similar to previous changes (6)

• plugins/huggingface/vision_agents/plugins/huggingface/transformers_vlm.py
• CHANGELOG.md
• tests/test_observability.py
• agents-core/vision_agents/core/agents/agents.py
• agents-core/vision_agents/core/observability/agent.py
• plugins/nvidia/vision_agents/plugins/nvidia/nvidia_vlm.py

Included review availability: Your plan includes up to 8 reviews per rolling hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

The change expands AgentMetrics with error, turn, realtime, and VLM fields. Average values now preserve sample counts during serialization. Realtime sessions record duration and time to first audio. TTS records total synthesis time and time to first audio separately. VLM providers report time to first token. Agent shutdown emits a final call_metrics_summary event before teardown. Tests cover the new metrics and inference modes.

Merge Risk: 🟡 Moderate · up to a1c04

The PR adds richer metrics and a final call summary, but two current issues can leave TransformersVLM time-to-first-token data missing and allow an outdated periodic event after the final summary, resulting in incomplete or inconsistent metrics for consumers.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

tests/test_observability.py (1)

55-76: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Remove mock-based OpenTelemetry assertions.

Remove mock_metrics and all assert_called_once_with checks. Assert AgentMetrics state in unit tests. Test root-only OpenTelemetry emission with a real exporter in an @pytest.mark.integration test.

Sources: Coding guidelines, Path instructions

plugins/huggingface/vision_agents/plugins/huggingface/transformers_vlm.py (1)

246-250: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Populate TTFT before forwarding it.

TransformersVLM.create_response() never sets LLMResponseFinal.time_to_first_token_ms. The final response is emitted without this field at Line 367-373, and the only delta sets it to None at Line 365. The collector therefore records no TTFT samples for Transformers inference. Measure the first token during generation and set the value on the final response, or remove this provider from the TTFT metric claim.

agents-core/vision_agents/core/agents/agents.py (1)

804-810: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Stop the metrics broadcast task before sending the final summary.

_metrics_broadcast_task is cancelled only after _emit_call_metrics_summary(). If the broadcast loop is already sending an agent_metrics event, that event can arrive after call_metrics_summary. Cancel and await the broadcast task before emitting the final summary.

Proposed ordering fix

-        # Emit final call metrics while the edge connection is still up.
-        await self._emit_call_metrics_summary()
-
         # Stop metrics broadcast task
         if self._metrics_broadcast_task:
             await cancel_and_wait(self._metrics_broadcast_task)
             self._metrics_broadcast_task = None
+
+        # Emit final call metrics while the edge connection is still up.
+        await self._emit_call_metrics_summary()

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: fe64ebd3-89c1-4426-a28f-c0992bf93452

📥 Commits

Reviewing files that changed from the base of the PR and between adc3523 and 70a74a1.

📒 Files selected for processing (20)

• CHANGELOG.md
• agents-core/vision_agents/core/agents/agents.py
• agents-core/vision_agents/core/llm/realtime.py
• agents-core/vision_agents/core/observability/__init__.py
• agents-core/vision_agents/core/observability/agent.py
• agents-core/vision_agents/core/observability/collector.py
• agents-core/vision_agents/core/observability/metrics.py
• agents-core/vision_agents/core/tts/events.py
• agents-core/vision_agents/core/tts/tts.py
• examples/01_simple_agent_example/simple_agent_example.py
• plugins/gemini/vision_agents/plugins/gemini/gemini_vlm.py
• plugins/huggingface/vision_agents/plugins/huggingface/huggingface_vlm.py
• plugins/huggingface/vision_agents/plugins/huggingface/transformers_vlm.py
• plugins/moondream/vision_agents/plugins/moondream/vlm/moondream_cloud_vlm.py
• plugins/moondream/vision_agents/plugins/moondream/vlm/moondream_local_vlm.py
• plugins/nvidia/vision_agents/plugins/nvidia/nvidia_vlm.py
• plugins/openai/vision_agents/plugins/openai/chat_completions/chat_completions_vlm.py
• plugins/twelvelabs/vision_agents/plugins/twelvelabs/pegasus_vlm.py
• tests/test_observability.py
• tests/test_tts_base.py

Included review availability: Your plan includes up to 8 reviews per rolling hour; 7 remain after this review.