Propose PR #807 conflict resolution on the dedicated runner - #808
Merged
Wenjie Fan (gggdttt) merged 8 commits intoAug 20, 2026
Merged
Conversation
Co-authored-by: wenjiefan <wenjiefan@microsoft.com> Copilot-Session: 045bef88-d7d4-4553-a681-d7d73cdb2285
Co-authored-by: wenjiefan <wenjiefan@microsoft.com>
Co-authored-by: wenjiefan <wenjiefan@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 003aa3eb-6da6-49f0-bb28-49c8bb0e0d3f
Record wall-clock runtime and structural BCQuality knowledge counts for dedicated PR review runs. Surface them in code-review summaries and the leaderboard without parsing console usage output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the dedicated production PR review runner while retaining structured performance metrics available from the filter report. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Parse the pinned BC-ALAgents schema-v1 OTel artifact strictly, retain exact usage and completeness data, and expose code-review performance aggregates without transcript parsing or runtime provenance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 15d832b0-a404-440c-932e-186ceebbb9a7
Wenjie Fan (gggdttt)
added a commit
to microsoft/BC-ALAgents
that referenced
this pull request
Aug 20, 2026
## Summary - enable the Copilot CLI OpenTelemetry file exporter for every generate/all invocation - aggregate raw `chat` spans into a versioned `_run-metrics.json` artifact in `REVIEW_OUTPUT_DIR` - remove transcript/process-log parsing and conversational turn-count proxies - keep the no-AL fast path on the same schema and isolate/delete raw OTel JSONL after harvesting - preserve existing review behavior and authentication forwarding This provides the upstream structured contract needed by [microsoft/BC-Bench#807](microsoft/BC-Bench#807) and [microsoft/BC-Bench#808](microsoft/BC-Bench#808). ## CLI version dependency BC-Bench #808 currently runs `npm install -g @github/copilot@1.0.80`, but that package version is not published: a clean isolated install returns npm `E404`/`ETARGET`, and the highest published stable version is `1.0.79`. Exact `1.0.80` validation is therefore impossible. The smallest explicit dependency fix is to pin #808 to `@github/copilot@1.0.79`, which this PR validates end to end with auto-update disabled. No transcript fallback is added. ## Structured source and schema The source is the Copilot CLI OTel JSONL file exporter: - `COPILOT_OTEL_EXPORTER_TYPE=file` - `COPILOT_OTEL_FILE_EXPORTER_PATH=<unique system-temp path>` - `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false` The engine aggregates individual spans whose `gen_ai.operation.name` is `chat` and deliberately ignores cumulative OTel metric snapshots. Schema version 1 always has these 18 fields: ```json { "schema_version": 1, "metrics_source": "copilot-cli-otel | not-applicable", "cli_version": "string | null", "wall_time_seconds": "number | null", "prompt_tokens": "integer | null", "cached_tokens": "integer | null", "cache_creation_tokens": "integer | null", "completion_tokens": "integer | null", "reasoning_tokens": "integer | null", "total_tokens": "integer | null", "api_calls": "integer | null", "failed_api_calls": "integer | null", "usage_api_calls": "integer | null", "ai_credits": "number | null", "premium_requests": "number | null", "models": ["string"], "usage_complete": "boolean", "malformed_records": "integer" } ``` Semantics: - `api_calls` is the raw valid `chat` span count, including nested agents, failures, and retries; it is never derived from conversation turns. - token fields sum `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, optional cache attributes, and optional `gen_ai.usage.reasoning.output_tokens`. `total_tokens` is input plus output because reasoning is an output subtype. - `ai_credits` is the exact sum of `github.copilot.nano_aiu / 1_000_000_000`. - `premium_requests` is the exact sum of `github.copilot.cost`, which the Copilot structured usage contract defines as the per-call premium-request multiplier. - credit/request totals are `null` unless every counted span exposes the corresponding source attribute; `usage_complete` is false if any counted span lacks input/output usage. - `malformed_records` counts invalid JSON lines and `chat` spans with invalid numeric or status attributes. Each malformed span is skipped independently so valid spans still produce an artifact and accurate valid-record totals. - `metrics_source=not-applicable` means the local no-AL preflight launched no Copilot process: exact use fields are zero, while provider-dependent `reasoning_tokens` and legacy `premium_requests` are null. - raw JSONL is never created under or copied to `REVIEW_OUTPUT_DIR`; timed-out processes are killed and awaited boundedly before disposal and harvest, deletion is retried for transient file locks, and parsed records are retained only in memory so later wall-time updates cannot erase or double-count usage. ## Source evidence A clean isolated `@github/copilot@1.0.79` install, invoked through its exact native binary with `--no-auto-update`, reported both CLI and OTel agent version `1.0.79`. A controlled tool-using prompt emitted two real `chat` spans. The production parser harvested exactly: - 2 API calls - 10,042 input tokens - 140 output tokens - 7 reasoning tokens - 4,939 cache-read tokens - 5,099 cache-creation tokens - 1.51433 AI credits - 2 premium requests The source JSONL was absent after aggregation, and no raw OTel file existed in the output directory. ## Validation - all PowerShell files parse successfully - Pester: 119 passed - configured PSScriptAnalyzer: 0 errors, 0 warnings - exact Copilot CLI `1.0.79` isolated OTel invocation and production-parser harvest: passed --------- Co-authored-by: wenjiefan <wenjiefan@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d9812db4-622a-447b-bc05-3025f0eecd10
Wenjie Fan (gggdttt)
marked this pull request as ready for review
August 20, 2026 13:48
Pin the merged BC-ALAgents metrics producer and published Copilot CLI, accept the canonical no-AL artifact, and retain nullable reasoning and premium usage in code-review results and dashboards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 15d832b0-a404-440c-932e-186ceebbb9a7
Wenjie Fan (gggdttt)
marked this pull request as draft
August 20, 2026 14:02
Wenjie Fan (gggdttt)
marked this pull request as ready for review
August 20, 2026 14:06
Wenjie Fan (gggdttt)
changed the base branch from
perf-eval-engine-metrics
to
main
August 20, 2026 14:21
Wenjie Fan (gggdttt)
changed the title
Wenjie Fan (gggdttt)
changed the base branch from
main
to
perf-eval-engine-metrics
August 20, 2026 14:22
Wenjie Fan (gggdttt)
changed the title
Wenjie Fan (gggdttt)
merged commit Aug 20, 2026
c0eabd1
into
perf-eval-engine-metrics
13 checks passed
Wenjie Fan (gggdttt)
deleted the
gggdttt-review-pr-807-conflict-resolution
branch
August 20, 2026 14:29
Wenjie Fan (gggdttt)
pushed a commit
that referenced
this pull request
Aug 20, 2026
Preserve the reviewed conflict-resolution tree while recording main as merged after PR #808 was squash-merged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 003aa3eb-6da6-49f0-bb28-49c8bb0e0d3f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This is a review-only proposal for Dayland. It targets
perf-eval-engine-metrics, the head branch of #807, notmain, so the semantic conflict resolution can be reviewed before any decision to incorporate it into #807.The proposal ports the performance telemetry onto the dedicated PR review runner accepted in #800. It restores the original usage fields that now have a stable structured contract, but intentionally does not claim full feature equivalence with the original #807 implementation.
Structured metrics dependency
This proposal depends on merged microsoft/BC-ALAgents#58 and pins its immutable merge commit
533dd39dfe29218c09e5e31c39c78bb72fa20aa2. The evaluation action pins the published and validated@github/copilot@1.0.79package.The production local/
allpath emitsREVIEW_OUTPUT_DIR/_run-metrics.json. BC-Bench accepts only the exact 18-field schema version 1 contract withmetrics_sourceset tocopilot-cli-otelor the canonical no-ALnot-applicableshape. Missing, malformed, wrong-version, wrong-type, extra-field, and noncanonical no-AL artifacts fail explicitly; there is no console transcript or process-log fallback.Scope comparison
_filter-report.jsoninput.ai_credits, and exactpremium_requests. Nullable producer values remain null; none are invented.For partial structured usage (
usage_complete: falseormalformed_records > 0), the proposal preserves every available exact count and persists both completeness and malformed-record metadata. Summary and dashboard completeness treat either condition as incomplete, so partial totals are never presented as complete.cli_versionandmodelsare validated as part of the producer schema but remain in the raw engine artifact rather than reintroducing runtime provenance into BC-Bench configuration or dashboard presentation.Dayland: please review whether this scope split preserves the intent of #807 and the architecture accepted in #800.
Validation
ty check: passedpre-commit run --all-files: passed