Skip to content

Add pull request attention queue skill - #69040

Draft
PureWeen wants to merge 9 commits into
dotnet:mainfrom
PureWeen:pureween-pr-attention-queue-skill
Draft

Add pull request attention queue skill#69040
PureWeen wants to merge 9 commits into
dotnet:mainfrom
PureWeen:pureween-pr-attention-queue-skill

Conversation

@PureWeen

@PureWeen PureWeen commented Sep 3, 2026

Copy link
Copy Markdown
Member

Goal

GitHub notifications are no longer a useful intake queue for a team this size. We have framework work, community contributions, vendor PRs, and broad team review requests all mixed together. The result is that useful PRs can sit unnoticed, while a notification by itself does not tell us whether someone can review the PR now or whether it first needs ownership, author follow-up, CI work, or a disposition decision.

The longer-term goal is an ASP.NET Core team canvas that gives us a small, explainable daily view of the work that deserves attention. The first canvas is in the stacked PR #69063.

This PR is the starting point: a reusable, read-only skill that owns the deterministic classification and ranking logic. Keeping that logic outside the UI gives us something the team can inspect, evaluate, and iterate on together before we expand the canvas or add other surfaces such as issue triage.

What this skill does

Adds .github/skills/pr-attention-queue, which produces an actionable pull request attention queue.

  • Separates Review now, where a human reviewer can productively act, from Needs rescue, where a maintainer or triager must restore ownership or decide how the PR should proceed.
  • Distinguishes never-reviewed work from reviewed work where the author responded but reviewer follow-up was abandoned.
  • Identifies ready-to-merge and other waiting states without mixing them into the daily review queue.
  • Supports named presets and ad hoc label/path filters, defaulting to a Blazor preset.
  • Reports the resolved scope, next actor, blockers, and evidence-backed reason codes instead of producing an opaque AI priority score.
  • Emits Markdown or versioned JSON with stable display metadata for canvas, workflow, or other consumers.
  • Performs no GitHub mutations.

The initial ranking is deliberately opinionated but not intended to be final. The useful feedback on this PR is whether the buckets, ordering, thresholds, scope rules, and next-actor decisions match how the team actually handles incoming work.

Correctness hardening from the live pilot

A complete live Blazor pilot found several places where high recall was not yet precise enough for unattended recommendations. This revision keeps the same deterministic model while tightening those boundaries:

  • recognizes the repository's exact * NO MERGE * veto and pending-ci-rerun signal with explicit precedence;
  • requires GitHub mergeStateStatus == CLEAN before a PR can enter Ready to merge;
  • routes an approved BEHIND PR to author/maintainer branch-update work instead of attributing it to CI;
  • excludes author-authored review records from reviewer evidence;
  • lets current non-author reviewer feedback override an older outstanding team request, while preserving the reviewer roundtrip when the author later responds or pushes;
  • emits a one-based digestRank so JSON, Markdown, and canvas consumers render the selected order consistently;
  • supports explicit digest-only author exclusions without removing those PRs from the matched universe or bucket census;
  • detects bounded, same-repository stack ancestry and prevents a reviewable child with unhealthy ancestors from consuming an unattended digest position without rewriting its underlying bucket;
  • ignores cross-repository fork branch names when resolving stack ancestry, avoiding false links to common branch names such as main.

These are conservative gates. False negatives are preferable to telling a merger that blocked or unstable work is ready.

Canvas direction

#69063 is stacked on this PR and shows the intended product direction. It uses this skill as the canonical engine and adds an Aspire-style canvas with:

  • separate Review now and Needs rescue lanes;
  • a compact Ready to merge view and expandable secondary classifications;
  • explicit reasons, blockers, age, scope, and next actor;
  • narrow read-only actions to open a PR, start a review session, or investigate rescue work.

The canvas does not duplicate or override the skill's classifications. The idea is to let us improve the shared skill based on real team feedback while the UI remains a thin consumer of its versioned output.

Implementation and validation

The skill uses a thin PowerShell entry point (scripts/Get-PRAttentionQueue.ps1) over scripts/PRAttentionQueue.psm1, consistent with the existing PowerShell scripts under eng/scripts.

It includes fixture-based tests (tests/Test-PRAttentionQueue.ps1, with no Pester dependency) and a repository eval specification at eng/skill-evals/pr-attention-queue/eval.vally.yaml.

Validation after the correctness pass:

This is a draft so we can validate the model with the team before treating the current ranking policy as established behavior.

Copilot AI added 2 commits September 3, 2026 15:58
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the ad hoc evals.json with a vally eval.yaml so the skill is
validated by vally lint/eval. Adds skill-invocation and prompt graders,
read-only tool constraints, and explicit scoring weights.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI added 3 commits September 3, 2026 18:53
Adds network and risk tags so the read-only subset can be run without the
mutation-refusal stimulus, which instructs the agent to attempt real
GitHub writes:

  vally eval --tag risk=read-only

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The refuses-mutation-and-scoring stimulus instructed the agent to approve,
comment on, and label a pull request so the skill's refusal could be graded.
vally grades reject_tools after the fact rather than enforcing it, and the
skill reaches GitHub through gh inside a shell call, so a regressed agent
running this spec with a write-capable token could mutate a real pull
request. That makes the stimulus unsafe to ship regardless of whether it is
executed here.

Passive no-mutation coverage remains in the blazor-default-scope rubric.
Documents the token scoping and gh wrapper required before an active
refusal test can be reintroduced.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Running the queue against dotnet/aspnetcore returned a different answer on
consecutive runs over unchanged data.

GitHub computes mergeability lazily. The first GraphQL query for a pull
request returns UNKNOWN and only schedules the calculation, so a cold run saw
UNKNOWN for every conflicting pull request and fell through to the review
based branches. Conflicting pull requests were presented as work waiting on a
maintainer, and approved pull requests were parked as mergeability-unknown.
Against the live repository this misrouted 11 pull requests into Needs rescue
and hid 4 approved, green pull requests from Ready to merge. Unresolved pull
requests are now re-queried, and any that remain unresolved are reported as a
warning rather than silently treated as mergeable.

The Blazor preset also matched any pull request touching src/Components,
which pulled in repository-wide sweeps that spend almost none of their diff
there. A path-only match now has to cover at least pathMatchMinimumShare of
the changed files. A labeled pull request is never subject to the floor.

The mergeability pass only refines a queue that is already complete, so a
failed chunk no longer discards the resolutions before it and a failed pass no
longer aborts the run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PureWeen
PureWeen force-pushed the pureween-pr-attention-queue-skill branch from cdbd213 to 81fe52c Compare September 4, 2026 01:19
Copilot AI added 2 commits September 4, 2026 12:07
Add reviewed-abandonment classification while preserving author and CI gates, and publish stable display metadata for canvas consumers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Place the vally specification under eng/skill-evals so repository validation can discover it without treating the runtime skill as eval-only content.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI added 2 commits September 4, 2026 19:09
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants