Create validated KBEs for new test quarantines

[PureWeen]

Note

This is ready for workflow-safety review. The implementation is complete, but automatic Known Build Error enrollment is still off until we do a separately approved post-merge canary.

What this unlocks

The test-quarantine workflow already collects the exact failing test, build, test result, error, and stack trace. This PR keeps that evidence and uses it to safely produce the full quarantine handoff for one eligible new flaky test:

1. A tracking issue with validated Build Insights JSON.
2. A short investigation comment.
3. The normal draft quarantine PR linked to that issue.

The useful part here is that we are not trusting the agent to decide whether a test is eligible or to render the final issue. The repository proves eligibility, the agent proposes a narrowly scoped matcher, and repository code validates everything again before anything is written.

You can see it working

I ran the complete live path in my public fork with synthetic failure data. The input is intentionally fake and clearly labeled, but the issue, comment, PR, temporary-ID resolution, threat detection, and safe-output processing are real:

• Successful workflow run
• Generated quarantine issue, including the four-key Build Insights JSON
• Generated investigation comment
• Generated draft quarantine PR

That run created exactly one issue, one comment, and one draft PR. aw_synthdemo resolved to issue 55 in the comment target, PR body, and committed [QuarantinedTest("https://github.com/PureWeen/aspnetcore/issues/55")] attribute. The PR targets this feature branch and changes only AdaptiveCapacityDictionaryTests.cs.

No ASP.NET Core test actually failed for this demo. No upstream state was changed, no Known Build Error label was applied, and the rollout variable remains disabled.

How the safety boundary works

Existing CI/Helix evidence
        |
        v
Deterministic eligibility receipt
        |
        v
Agent proposes one matcher
        |
        v
Repository handler revalidates everything
        |
        +--> verified: issue with KBE JSON
        |
        +--> uncertain: ordinary test-failure issue, no KBE activation

The agent cannot choose the repository, issue title, final body, labels, build identity, eligibility result, or final JSON. It can only choose one collector-approved test and propose a matcher copied from the supplied failure evidence.

The collector requires an exact individual test, exact source resolution, at least two distinct post-cutoff failing builds, no consistent regression, no current method/type/inherited/assembly quarantine, no applicable quarantine-removal history, and an exact evidence build/run/result match. Missing or ambiguous evidence fails closed.

Important hardening from review

The review work found a few places where the first version was not conservative enough. The final branch now:

• Reconstructs assembly-level quarantine history across the whole test project, including separate files, deleted files, and renamed test sources.
• Handles inherited test runners conservatively and fails closed when project/history association is ambiguous.
• Directly covers Source B authentication, changed-file pagination, and rename handling.
• Independently limits the custom issue handler to one invocation.
• Uses strongly consistent issue listing plus exact-title matching instead of GitHub Search for deduplication.
• Keeps overlapping safe-output runs serialized without cancelling an in-progress write.

Repository commit 71e388eab781698fa8e4e8afa5f3df464788f4c1, which removed an assembly quarantine from a separate ProjectTemplates fixture file, is the representative real history case used to verify the collector.

Rollout is deliberately separate

Every new issue receives test-failure. It receives Known Build Error only when every validation succeeds and TEST_QUARANTINE_ENABLE_KBE is exactly true.

This PR does not create or enable that variable. So merging this code does not enroll anything in the shared DNCENG Known Build Errors project. The first real enrollment remains a small, explicitly approved post-merge canary that we can turn off immediately.

The fork demonstration proves our repository and GitHub write path. It does not claim to prove DNCENG project enrollment or matching of a real Build Insights occurrence.

Validation

Production head: 48b6a87a99b363e681e0fb171421da301744863b

• Collector syntax and regression fixtures pass.
• The Node suite extracts and executes the actual inline safe-output handler, rather than testing a copy.
• Strict gh-aw v0.88.2 compilation and generated-workflow validation pass.
• Existing PR checks are green and the branch is mergeable.
• Eligible Case A staged run exercised the successful handler path without writes.
• Ineligible staged run proved that an already-quarantined test cannot emit KBE JSON or labeling.

Detailed eligibility and matcher rules

Deterministic Case A gates

The collector requires:

• an individual Source A/B test, not a work-item grouping;
• exact C# source resolution, including inherited tests reported as DerivedType.BaseMethod;
• no method, type, inherited runner-type, or assembly quarantine;
• no latest applicable quarantine-removal transition;
• Case A classification, never Case B;
• is_consistent_regression == false for Source A;
• at least two distinct failure builds strictly after the latest conservative cutoff;
• exact evidence build/run/result identity;
• exclusion of Source B PRs that touched a declaring or runner path, including previous_filename for renames.

The cutoff includes first-parent origin/main changes to every declaring/runner file, quarantine-removal history, and trusted prior suppression/attempt history. An unrelated same-file edit can defer a valid candidate, but stale evidence cannot activate a KBE.

Matcher rules

The local create-kbe skill follows dotnet/runtime matcher conventions and prefers:

1. An exact, case-sensitive literal failure message.
2. An ordered array of exact lines when one line is insufficient.
3. A bounded regex only when volatile content prevents a safe literal.

The handler rejects bare test names, stack frames, exception types, generic assertion/timeout/crash/infrastructure text, cross-test matches, unsafe regex constructs, and oversized bounds.

A verified issue ends with exactly one Build Insights block:

{
  "ErrorMessage": "...",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": true
}

Exactly one matcher field is populated. If eligibility, evidence identity, duplicate status, or matcher specificity cannot be proven, the handler creates the ordinary test-failure issue without KBE JSON or KBE labeling.

Operational boundaries

• One exact fully qualified test and at most one new Case A issue/PR/comment chain per activation.
• Threat detection must succeed before any write.
• Exact-title open issues are reused without editing or relabeling them.
• Human log excerpts are secret-scrubbed and HTML-escaped.
• Existing Case B re-quarantine and unquarantine behavior remains unchanged.
• This does not fix tests, remove quarantine, close issues, approve PRs, or merge anything.
• Generated .lock.yml changes remain compiler-owned under the repository-pinned gh-aw v0.88.2.

[PureWeen]

Three high-confidence correctness/reliability issues survived independent review and cross-checking. The action pin, generated lock consistency, temporary-ID wiring, staged handler contract, and output escaping did not produce actionable findings.