Add pull request attention queue skill - #69040
Draft
PureWeen wants to merge 9 commits into
Draft
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the ad hoc evals.json with a vally eval.yaml so the skill is validated by vally lint/eval. Adds skill-invocation and prompt graders, read-only tool constraints, and explicit scoring weights. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds network and risk tags so the read-only subset can be run without the mutation-refusal stimulus, which instructs the agent to attempt real GitHub writes: vally eval --tag risk=read-only Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The refuses-mutation-and-scoring stimulus instructed the agent to approve, comment on, and label a pull request so the skill's refusal could be graded. vally grades reject_tools after the fact rather than enforcing it, and the skill reaches GitHub through gh inside a shell call, so a regressed agent running this spec with a write-capable token could mutate a real pull request. That makes the stimulus unsafe to ship regardless of whether it is executed here. Passive no-mutation coverage remains in the blazor-default-scope rubric. Documents the token scoping and gh wrapper required before an active refusal test can be reintroduced. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Running the queue against dotnet/aspnetcore returned a different answer on consecutive runs over unchanged data. GitHub computes mergeability lazily. The first GraphQL query for a pull request returns UNKNOWN and only schedules the calculation, so a cold run saw UNKNOWN for every conflicting pull request and fell through to the review based branches. Conflicting pull requests were presented as work waiting on a maintainer, and approved pull requests were parked as mergeability-unknown. Against the live repository this misrouted 11 pull requests into Needs rescue and hid 4 approved, green pull requests from Ready to merge. Unresolved pull requests are now re-queried, and any that remain unresolved are reported as a warning rather than silently treated as mergeable. The Blazor preset also matched any pull request touching src/Components, which pulled in repository-wide sweeps that spend almost none of their diff there. A path-only match now has to cover at least pathMatchMinimumShare of the changed files. A labeled pull request is never subject to the floor. The mergeability pass only refines a queue that is already complete, so a failed chunk no longer discards the resolutions before it and a failed pass no longer aborts the run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PureWeen
force-pushed
the
pureween-pr-attention-queue-skill
branch
from
September 4, 2026 01:19
cdbd213 to
81fe52c
Compare
Add reviewed-abandonment classification while preserving author and CI gates, and publish stable display metadata for canvas consumers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Place the vally specification under eng/skill-evals so repository validation can discover it without treating the runtime skill as eval-only content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 4, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
GitHub notifications are no longer a useful intake queue for a team this size. We have framework work, community contributions, vendor PRs, and broad team review requests all mixed together. The result is that useful PRs can sit unnoticed, while a notification by itself does not tell us whether someone can review the PR now or whether it first needs ownership, author follow-up, CI work, or a disposition decision.
The longer-term goal is an ASP.NET Core team canvas that gives us a small, explainable daily view of the work that deserves attention. The first canvas is in the stacked PR #69063.
This PR is the starting point: a reusable, read-only skill that owns the deterministic classification and ranking logic. Keeping that logic outside the UI gives us something the team can inspect, evaluate, and iterate on together before we expand the canvas or add other surfaces such as issue triage.
What this skill does
Adds
.github/skills/pr-attention-queue, which produces an actionable pull request attention queue.The initial ranking is deliberately opinionated but not intended to be final. The useful feedback on this PR is whether the buckets, ordering, thresholds, scope rules, and next-actor decisions match how the team actually handles incoming work.
Correctness hardening from the live pilot
A complete live Blazor pilot found several places where high recall was not yet precise enough for unattended recommendations. This revision keeps the same deterministic model while tightening those boundaries:
* NO MERGE *veto andpending-ci-rerunsignal with explicit precedence;mergeStateStatus == CLEANbefore a PR can enter Ready to merge;BEHINDPR to author/maintainer branch-update work instead of attributing it to CI;digestRankso JSON, Markdown, and canvas consumers render the selected order consistently;main.These are conservative gates. False negatives are preferable to telling a merger that blocked or unstable work is ready.
Canvas direction
#69063 is stacked on this PR and shows the intended product direction. It uses this skill as the canonical engine and adds an Aspire-style canvas with:
The canvas does not duplicate or override the skill's classifications. The idea is to let us improve the shared skill based on real team feedback while the UI remains a thin consumer of its versioned output.
Implementation and validation
The skill uses a thin PowerShell entry point (
scripts/Get-PRAttentionQueue.ps1) overscripts/PRAttentionQueue.psm1, consistent with the existing PowerShell scripts undereng/scripts.It includes fixture-based tests (
tests/Test-PRAttentionQueue.ps1, with no Pester dependency) and a repository eval specification ateng/skill-evals/pr-attention-queue/eval.vally.yaml.Validation after the correctness pass:
BEHINDrouting, and digest controls;* NO MERGE *;pending-ci-rerun;UNSTABLErather thanCLEAN;PureWeenretained those PRs in the census while removing them from the visible digest;This is a draft so we can validate the model with the team before treating the current ranking policy as established behavior.