Add PR review performance telemetry and dashboards - #807
Open
dayland wants to merge 5 commits into
Open
Conversation
Collect BC-ALAgents usage and knowledge metrics, expose Code Review performance dashboards, and make engine and BCQuality experiment sources independently configurable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 20, 2026
Wenjie Fan (gggdttt)
added a commit
to microsoft/BC-ALAgents
that referenced
this pull request
Aug 20, 2026
## Summary - enable the Copilot CLI OpenTelemetry file exporter for every generate/all invocation - aggregate raw `chat` spans into a versioned `_run-metrics.json` artifact in `REVIEW_OUTPUT_DIR` - remove transcript/process-log parsing and conversational turn-count proxies - keep the no-AL fast path on the same schema and isolate/delete raw OTel JSONL after harvesting - preserve existing review behavior and authentication forwarding This provides the upstream structured contract needed by [microsoft/BC-Bench#807](microsoft/BC-Bench#807) and [microsoft/BC-Bench#808](microsoft/BC-Bench#808). ## CLI version dependency BC-Bench #808 currently runs `npm install -g @github/copilot@1.0.80`, but that package version is not published: a clean isolated install returns npm `E404`/`ETARGET`, and the highest published stable version is `1.0.79`. Exact `1.0.80` validation is therefore impossible. The smallest explicit dependency fix is to pin #808 to `@github/copilot@1.0.79`, which this PR validates end to end with auto-update disabled. No transcript fallback is added. ## Structured source and schema The source is the Copilot CLI OTel JSONL file exporter: - `COPILOT_OTEL_EXPORTER_TYPE=file` - `COPILOT_OTEL_FILE_EXPORTER_PATH=<unique system-temp path>` - `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false` The engine aggregates individual spans whose `gen_ai.operation.name` is `chat` and deliberately ignores cumulative OTel metric snapshots. Schema version 1 always has these 18 fields: ```json { "schema_version": 1, "metrics_source": "copilot-cli-otel | not-applicable", "cli_version": "string | null", "wall_time_seconds": "number | null", "prompt_tokens": "integer | null", "cached_tokens": "integer | null", "cache_creation_tokens": "integer | null", "completion_tokens": "integer | null", "reasoning_tokens": "integer | null", "total_tokens": "integer | null", "api_calls": "integer | null", "failed_api_calls": "integer | null", "usage_api_calls": "integer | null", "ai_credits": "number | null", "premium_requests": "number | null", "models": ["string"], "usage_complete": "boolean", "malformed_records": "integer" } ``` Semantics: - `api_calls` is the raw valid `chat` span count, including nested agents, failures, and retries; it is never derived from conversation turns. - token fields sum `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, optional cache attributes, and optional `gen_ai.usage.reasoning.output_tokens`. `total_tokens` is input plus output because reasoning is an output subtype. - `ai_credits` is the exact sum of `github.copilot.nano_aiu / 1_000_000_000`. - `premium_requests` is the exact sum of `github.copilot.cost`, which the Copilot structured usage contract defines as the per-call premium-request multiplier. - credit/request totals are `null` unless every counted span exposes the corresponding source attribute; `usage_complete` is false if any counted span lacks input/output usage. - `malformed_records` counts invalid JSON lines and `chat` spans with invalid numeric or status attributes. Each malformed span is skipped independently so valid spans still produce an artifact and accurate valid-record totals. - `metrics_source=not-applicable` means the local no-AL preflight launched no Copilot process: exact use fields are zero, while provider-dependent `reasoning_tokens` and legacy `premium_requests` are null. - raw JSONL is never created under or copied to `REVIEW_OUTPUT_DIR`; timed-out processes are killed and awaited boundedly before disposal and harvest, deletion is retried for transient file locks, and parsed records are retained only in memory so later wall-time updates cannot erase or double-count usage. ## Source evidence A clean isolated `@github/copilot@1.0.79` install, invoked through its exact native binary with `--no-auto-update`, reported both CLI and OTel agent version `1.0.79`. A controlled tool-using prompt emitted two real `chat` spans. The production parser harvested exactly: - 2 API calls - 10,042 input tokens - 140 output tokens - 7 reasoning tokens - 4,939 cache-read tokens - 5,099 cache-creation tokens - 1.51433 AI credits - 2 premium requests The source JSONL was absent after aggregation, and no raw OTel file existed in the output directory. ## Validation - all PowerShell files parse successfully - Pester: 119 passed - configured PSScriptAnalyzer: 0 errors, 0 warnings - exact Copilot CLI `1.0.79` isolated OTel invocation and production-parser harvest: passed --------- Co-authored-by: wenjiefan <wenjiefan@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9812db4-622a-447b-bc05-3025f0eecd10
Co-authored-by: wenjiefan <wenjiefan@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: dayland <dayland@microsoft.com> Copilot-Session: 045bef88-d7d4-4553-a681-d7d73cdb2285 Copilot-Session: 003aa3eb-6da6-49f0-bb28-49c8bb0e0d3f Copilot-Session: 15d832b0-a404-440c-932e-186ceebbb9a7
Preserve the reviewed conflict-resolution tree while recording main as merged after PR #808 was squash-merged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 003aa3eb-6da6-49f0-bb28-49c8bb0e0d3f
Wenjie Fan (gggdttt)
previously approved these changes
Aug 20, 2026
Sun Haoran (haoranpb)
requested changes
Aug 21, 2026
Sun Haoran (haoranpb)
left a comment
Sun Haoran (haoranpb)
left a comment
Collaborator
There was a problem hiding this comment.
In general, I think this is great and fills a gap. The PR Review Agent currently gives us very little performance data beyond wall time, meaning we can't compare with baseline (using Copilot CLI) for performance metrics.
My main concern is that we may be promoting every available signal into the result and leaderboard schemas without a clear decision we expect each metric to support. For the headline comparison, I think the useful set is quality/F1, wall time, and AI credits, potentially with valid-output rate. The token breakdown, API/failed/usage-call counts, completeness, and malformed-record fields feel more like diagnostics that could remain in the raw artifact or CI summary.
| api_calls: int | None = Field(default=None, ge=0, exclude_if=lambda value: value is None) | ||
| failed_api_calls: int | None = Field(default=None, ge=0, exclude_if=lambda value: value is None) | ||
| usage_api_calls: int | None = Field(default=None, ge=0, exclude_if=lambda value: value is None) | ||
| ai_credits: float | None = Field(default=None, ge=0, exclude_if=lambda value: value is None) |
Collaborator
There was a problem hiding this comment.
Could you merge/rebase main and resolve ai_credits here? #810 already added this field.
Comment on lines
+96
to
+97
| knowledge_files: int | None = Field(default=None, ge=0, exclude_if=lambda value: value is None) | ||
| knowledge_pruned: int | None = Field(default=None, ge=0, exclude_if=lambda value: value is None) |
Collaborator
There was a problem hiding this comment.
[design] Reading the code that calculate these two fields, are they not deterministic basing on the commit sha from BCQuality? If so, recording that commit sha is enough to track those metrics?
You can then move the code to conduct analysis in a notebook or something.
| prompt_tokens: int | None = None | ||
| completion_tokens: int | None = None | ||
|
|
||
| # Structured usage metrics emitted by agent harnesses |
Collaborator
There was a problem hiding this comment.
All those metrics are calculated by reading a file PR Review Agent left behind, so it is not entirely clear what each of those mean: e.g. usage_complete and malformed_records.
Add comments to elaborate.
Also, some metrics might be able to consolate. e.g. is turn_count and api_calls the same?
Preserve the original public performance metrics while keeping later producer diagnostics out of persisted results and dashboards. Gate promoted usage values on complete, well-formed telemetry and retain exact AI credit precision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds performance telemetry for the dedicated BC PR Review evaluation runner without replacing the engine's raw diagnostic artifact.
Changes
_run-metrics.jsonartifact, including source, field presence, types, and the canonical no-AL shape.AgentMetrics.ai_creditsfield introduced onmainand preserves full precision through summary persistence.533dd39dfe29218c09e5e31c39c78bb72fa20aa2.@github/copilotto1.0.79:1.0.80is not published to npm, and1.0.79is the version whose structured telemetry contract was validated.Validation
ty check . --ignore=unresolved-import --exclude notebooks/pre-commit run --all-filesAB#645219