Pedroro/release scout - #457
Draft
pedro romero vargas (p3dr0rv) wants to merge 83 commits into
Draft
Conversation
… projects - Introduced `checks.py` in the `release-agent/tools` directory, implementing various readiness checks including Azure DevOps build definition access, HTTP reachability, and pipeline variable management. - Updated `settings.gradle` to comment out several project inclusions, including `AcaPlugin`, `LinuxBroker`, `java-linux-test-app`, `LinuxBrokerPackage`, and `NativeAuthSample`, to streamline the build configuration.
…e conditions and ensure safe parallel execution; enhance error messaging for Azure DevOps authentication issues.
|
❌ Work item link check failed. Description does not contain AB#{ID}. Click here to Learn more. |
Engine (gate integrity): - sign: remove blanket --all; CLI now requires explicit --item ids + refuses a bare sign, and records each attestation individually with an evidence --note. Closes the hole where sign --all attested every human item in one blind call (a release reached Phase 0 without real confirmation). - readiness.sign() carries a per-item evidence note. Readiness config/UX: - silent_perms.required_servers now includes kusto + icm (the MCP servers the adx_access / oncall_now checks call) so they don't prompt on first run. Skill (A: dedupe, B: modular split): - Split the 48KB monolith SKILL.md into a lean 7KB core (golden rules + behaviour dispatch + reference routing table) plus skill/reference/*.md read on demand (readiness-gate, starting-and-scheduling, presenting-status, commands, phases/preflight, phases/_TEMPLATE). Core is now well under Scout's inline limit, so it no longer spills / gets skimmed. - Fix regression where the readiness table was suppressed: render the checklist table FIRST, then handle silent-runs (was "before showing the checklist"). - Anti-assumption rule: never attest/approve on an echoed m_ask_user result. - Scale convention: add a phase = phases.yaml + phases/agents/<id>.py + reference/phases/<id>.md + one routing-table row. bootstrap.ps1: auto-install Python via winget + silent pyyaml; verify Scout present + skill copy; restart Scout by default (guarded when run inside a Scout session); UTF-8 + detached-launch fixes; clearer folder/run instructions. Tests: 101/101 (added regressions for bare-sign refusal, no --all, evidence note). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recurring regression: the model ran checklist (which auto-logs) but did not paste the table into its reply, then asked for attestations in a table-less message — losing the context the user needs. - Core golden rule #2: running a command != showing it; the user only sees what you paste. - New golden rule #2b: NEVER ask for a gate decision or attestation in a message that doesn't contain the freshly-rendered table (in the SAME message). - readiness-gate.md step 3b: paste the table verbatim into the same message as the attestation question. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ecks Two UX nits from testing: 1. Table rendered twice (once mid-auto-checks, once after). Restructured the readiness flow to render the checklist table ONCE, in step 3b, right before the attestations — steps 1-3a are now quiet setup (verify + silent-runs + scout checks) with no half-done table. Step 1 uses �erify (terse pass/fail) instead of checklist --verify (big table). 2. Model asked the user permission to run the ADX/Kusto query even though the kusto MCP server is auto-approved (verified: the query runs silently). The prompt was the model asking, not Scout. Step 3a now explicitly forbids asking permission to run the scout-assisted auto checks — they're auto-approved and must run silently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… auto checks" This reverts commit b8cc7b7.
…uctural fix) Prompt-level 'render the table then ask' instructions were fragile — the model would run checklist (auto-logged) but not paste it, then ask attestations in a table-less message. Fix structurally in the engine instead: - render.readiness_table now appends a '✋ Your confirmation needed' section listing the outstanding attest items, but ONLY once every auto item is satisfied. The table and the attestation prompt are now a single deterministic output — you can't show one without the other, and the prompt can't appear before the auto checks are done. Engine controls the ordering, not the model. Skill readiness flow simplified to match: run all auto checks silently first (never ask permission to run the ADX/on-call MCP calls — they're auto-approved), then render checklist ONCE (its output carries the confirmation section) and fire m_ask_user in the same message. Fixes both nits: single table render, and no spurious 'can I run the query?' prompt. Tests: 102/102 (added test_attest_prompt_welded_only_after_auto_done). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…put (structural fix)" This reverts commit c8c0e13.
…plays) Root-cause of the recurring 'no table, just a list' regression: it's controlled by WHERE the checklist render sits in the tool-call sequence, not by wording. - A checklist table render that is the model's FIRST action is pasted verbatim. - A render placed AFTER a chain of silent tool calls (verify/record-check/ m_get_settings) gets summarized into a plain list (model switches to wrap-up mode). The first render also primes verbatim-pasting for the second. So the double-render is the control mechanism, not waste. Codified as a DESIGN INVARIANT in readiness-gate.md so future edits don't collapse to a single end-of-flow render and reintroduce the regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…wo-render flow) Now that we understand the render-first mechanism, re-apply the engine welded prompt WITHOUT the flow reorder that broke display: - render.readiness_table appends '✋ Your confirmation needed' only once all auto items are satisfied. In the working two-render flow this means render #1 (checklist --verify, scout items still pending) stays a clean table with NO prompt, and render #2 (after the silent scout checks) carries the welded attestation list. Verified: render1 no prompt, render2 has prompt. - Keeps the proven display-first two-render flow (render #1 primes verbatim paste); adds deterministic attestation context to render #2 from the CLI so the model doesn't hand-build the list. - step 3b notes the confirmation section comes from the CLI. Tests: 102/102. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lock User wants the table shown once, not twice. The two outputs are inherent (one before the silent auto-checks, one after) — the fix is to make the SECOND output a compact confirmation block, not the full table again. - render.py: full table no longer carries the welded prompt; new public render.attest_prompt(chk) returns just the '✋ Your confirmation needed' block (or a status line if auto checks aren't done / gate cleared). - new CLI mode: checklist --attest-prompt emits ONLY that block (no table). - Flow: render #1 = full table (display-first, primes verbatim paste); silent auto checks; render #2 = checklist --attest-prompt = block only. One table total. Verified E2E: render1 has table/no block, render2 has block/no table. - Invariant doc + step 3b updated to the single-table design. Tests: 102/102. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mplete table once + deterministic attestation card
Fixes the recurring readiness-display problems with a two-part approach:
1. Deterministic attestation card (removes the LLM from the critical path).
- render.attest_prompt_payload(chk, release) builds the exact m_ask_user
payload the engine owns: question + answer cards (1 confirm_all + one
decline per outstanding attest item) + confirm_items + recommendedIndex.
ready:false until all auto checks pass. On-call window dates surfaced in
the confirm_all description.
- `checklist --attest-prompt --json` returns that payload; the skill passes
it straight to m_ask_user. The always-rendered Scout card is the guaranteed
source of truth even if the markdown table doesn't render.
2. Table shown ONCE, AFTER full evaluation (the requested UX).
- Skill flow reordered: steps 1-3 run all auto checks SILENTLY (Python
verifiers, silent_perms, on-call, ADX) with no table/partial renders; step
4 renders the complete checklist table (all [auto] items resolved to the
final state) with explicit "paste verbatim, do not summarize" framing;
step 5 presents the deterministic card; steps 6-8 map the answer to
sign/decline. No more early/incomplete table, no spurious ADX permission
prompt.
Gate integrity preserved throughout: no --all, explicit per-item sign, never
attest on assumption.
Tests: 103/103 (added test_attest_prompt_payload_is_deterministic_card,
test_attest_prompt_is_separate_render_never_in_table).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The phase map and current-phase steps tables used an empty-header first column just for the status icon (| | # | Phase | Done |). Scout's table renderer gives that empty column a huge width, leaving a big gap before the # column. Fix: drop the standalone icon column and prefix the icon onto the Phase/Step name — '| 0 | 🗓 Pre-flight … |' instead of '| 🗓 | 0 | Pre-flight … |'. Clean 3-/2-column tables, no empty gap. (Readiness table left as-is — user-validated.) Tests: 103/103. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Give every Phase-0 step ONE home under steps/<phase>/ authoring a single
build(state) -> Outcome, replacing the split where scout steps were scattered
across phases.yaml + stub_runner + commands/*.py + templates + skill md.
Outcomes (orchestrator/outcomes.py): Done | Blocked | NeedsHuman | NeedsSkill.
Generic dispatcher `step-action` serializes a migrated step's outcome as JSON so
the skill reacts uniformly (needs_skill -> run tool + record-step; needs_human ->
show prompt + done). Agent steps run in-process during `next` (step-action
refuses them).
Migrated (steps/preflight/):
scout : notice, flight_reminder, lockdown (gather-then-decide via check-lockdown)
attest: confirm_reminders, vitals (NeedsHuman prompts)
agent : breaking, cg, cron, wiki (Done/Blocked, engine-run)
Shared libs: steps/lib/{context,templating,agent}. Legacy homes
(phases/agents/preflight.py, commands/notice.py, commands/lockdown.py) reduced to
thin compatibility/recorder shims so the engine + 103 tests are unchanged.
Also fixes:
- render.py: phase-map legend was glued into the table as a one-column row
(missing blank line) -> now a separate italic caption.
- skill docs: render the status table ONCE per advance pass (after the work),
not an interim table before running the scout steps.
103/103 tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ils + knowledge
Replace the dry-run boolean with a real-by-default model plus a personal,
gitignored mocks.local.yaml overlay for safe testing:
- Remove dry_run entirely (state/engine/render/cli/steps). Every run makes real
calls; the fake "[dry-run] Would…" branches are gone.
- Mock overlay (orchestrator/mocks.py + steps/lib/mockctx.py): mocks.local.yaml
can, per step, force outcome (done|blocked), redirect a real send (send_to),
or inject inputs so the step's REAL logic runs on your data (cg alerts,
breaking changelog, cron run, wiki page_name/name_suffix, notice variant).
Readiness entry-gate auto checks are mockable too (readiness.<item>).
- Each step DECLARES its mockable knobs (MOCKABLE); `mock-spec` lists exactly
what every step exposes (steps + readiness), with typo detection.
Status "Details" column (render): a third column captures each step's execution
outcome from its stored note — where a lockdown clashed, the breaking change, CG
alerts found, the created wiki link — generic for all current/future steps, with
rich multi-line notes expanding below. Durable refs are now first-class:
StepState.links [{name,url}] (wiki page, CG alerts page + per-alert deep links),
threaded Done/Blocked -> StepResult -> StepState -> render.
Step knowledge base (config/knowledge.yaml + orchestrator/knowledge.py +
`step-info`): curated per-step help (what/where/how/links/faqs) so the skill
answers step questions accurately instead of guessing. Fixes the Play Console
vitals navigation. SKILL.md consults step-info for step questions.
Also: readiness table groups auto items before attest (oncall_now before
play_console_access); breaking-change entries no longer double their bullet.
106/106 tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…registries Make a step a single self-contained module so adding one touches ~2 files and can never silently drift: - Auto-discover step modules: steps/__init__.discover() scans steps/<phase>/*.py. Remove the hand-maintained _STEPS list. - Engine resolves agent steps via steps.get_step() (module.run when KIND==agent), stub fallback otherwise. DELETE phases/agents/ (package + REGISTRY) and the dead `agent:` field from config/phases.yaml — no runner registry, no agent-id indirection. - Co-locate step CONFIG into each module (CONFIG constant); remove config/preflight.yaml and orchestrator/phase_config.py. The module is now the single home for a step's logic + mock knobs + config. - Structural guardrail test: every discovered module maps to a phases.yaml step and its KIND matches the config flags — drift fails LOUDLY. config/phases.yaml is now pure COMPOSITION (which steps, order, deps, flags); a step's logic/config lives in its module; knowledge stays in config/knowledge.yaml (a module may override via KNOWLEDGE). README documents the 2-file "adding a step" contract. Tests point at steps.preflight.* directly; block-path tests use the mock overlay instead of registry patching. 107/107 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Correct the Component Governance alerts link to the real One/AD-MFA phonefactor-phoneApp-android componentGovernance page (in the cg module CONFIG and knowledge.yaml). - Pending scout steps (e.g. lockdown) now render as 'Scout runs this' instead of 'Your action' and are no longer flagged as needing the user; they only escalate to a blocked user action when a scout step records an attention/overlap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The knowledge base had no ownership fact for the CG step, so 'who fixes this?' was answered by guessing (wrongly deferring to the component owner). Record the real rule: the RELEASE OWNER remediates Component Governance alerts, including authoring and landing the fix PR. Add a 'who' field to the knowledge schema/renderer and populate it (plus a matching FAQ) for preflight.cg. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Free-form step questions (what/why/who/how about a step) are exactly the 'interaction' the event log is meant to capture, but nothing journaled them, so surfacing missing/inaccurate knowledge (e.g. CG ownership) left no trace. Add a thin two-sided 'step_qa' event: EventLog.qa(), a 'journal --kind qa --question ... --answer ...' path, log-viewer rendering, and a questions_answered rollup in summarize(). Wire the skill so that, during an active release, answering a step question from step-info silently journals the pair (skipped when no release run exists). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ocalization Build out the first three CCD steps as self-contained scout modules under steps/ccd/ (auto-discovered), each doing the real work idempotently: - final_reminder (email) — CCD-day 'code complete is TODAY' notice to the Android DL (the 'update' variant of the Phase-0 notice template). - pr_reminder (Teams) — NEW step; posts to the 'Code reviews' chat: merge every required PR before the 11 PM auto branch-cut; a CCD delay or post-CCD cherry-pick both need Moumita Ghosh's (moghosh@) approval; merge string PRs before the noon localization run. - localization (trigger)— fires pipeline 405133 (msazure/One) with isCreatePrSelected=true; PR created => strings to merge, no PR => none. Each module carries CONFIG (incl. fire_at_local timing), MOCKABLE knobs, knowledge.yaml entries, and tests. phases.yaml marks the three source: scout and inserts pr_reminder (P1-1b); branch_cut stays the human gate. Timing is encoded per-step (09:00 / 12:00) for a per-release CCD automation to drive; the engine stays date-based. EXTERNAL-REFERENCES catalogs the new chat, pipeline, and approver. Test harness gains _clear_ccd_scout so flow tests reach the branch_cut gate. 114/114. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire the timed Phase-1 automations into release-start provisioning, and close the traceability gap: the registry knew an automation's id/release/purpose but not WHICH STEPS it drives. - config/automations.yaml: declarative source of truth mapping each per-release automation to the steps it drives. Timing is DERIVED from each step module's fire_at_local (no duplication). - orchestrator/automations.py: plan(release, ccd) emits concrete specs (name/schedule/steps/prompt/registration) for the skill to create; validate() enforces every fire_at_local step is owned by exactly one automation and each automation's steps exist + share one fire time. - registry: entries now carry steps[]; list(step=...) is the reverse lookup (which automation owns a step). �utomation plan CLI action + --step / --step-filter flags. - Guardrail test test_automations_cover_every_scheduled_step makes the mapping self-enforcing; plus registry-linkage and plan tests. - SKILL/commands/scheduling docs: provision timed automations at start (register WITH --step), reverse/forward lookup, runtime journaling. 117/117. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
'Release push reminders' drives no steps — it advances the whole release via hourly tick. An empty steps[] read the same as 'not filled in', so label the category explicitly. Registry entries now carry a 'kind': step-driving (owns steps) or release-level (whole-release, no steps), auto-derived from whether steps are present, with a guard that the two can't contradict. kind_of() derives it for legacy entries. CLI: --kind on register (override) and list (filter); list shows [kind] and '(release-level - no steps)'. Docs note push reminders is release-level. 118/118. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Code Complete Day is now three scout steps (final_reminder, pr_reminder, localization) with NO gate: - branch_cut: removed — the release branch is cut AUTOMATICALLY at 11 PM, there is no manual cut action, so a human 'cut the branch' gate was wrong. - precheck_prs: removed — there's no reliable signal for which PRs are 'required' for a release, so an automated pre-check can't be built honestly. - verify_trigger: removed from Phase 1 — verifying the overnight/auto build fits Phase 2 (Build & Lib Verification), not Code Complete Day. Consequence: the first human gate is now go_test (Phase 2). Updated the flow tests (first gate go_test, second stop the bug-bash ui_failures hold), the phase-shape test, the mocks example, and stale comments. 118/118. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make localization a small state machine instead of fire-and-describe:
- TRIGGER: build() runs pipeline 405133 (isCreatePrSelected=true); the runner
calls record-localization-run to store the build id + start time and leaves
the step IN-FLIGHT (StepState gains a durable data dict).
- POLL: a per-release "every 10 minutes" poller automation calls
check-localization; the pure decide() applies:
* running and under 3h -> wait, poll again
* running and over 3h -> email the release engineer (with the manual-steps
localization doc) and hold the step
* complete -> parse the OneLocBuild@3 log for the
"Pull request created with ID" line; if found, post that PR to the Code
reviews chat for review and mark done (with the PR link); else done (no
strings).
All decision logic is pure/tested; IO is done via NeedsSkill/decision payloads.
Adds: steps/ccd/localization.py rewrite (helpers, decide, KNOWLEDGE incl. the
eng.ms localization doc), orchestrator/commands/localization.py
(record-localization-run, check-localization), StepState.data, an interval
"every N" automation kind in automations.yaml + planner/validate (pollers are
exempt from fire_at_local single-owner accounting; may share a step with a
time-of-day trigger), and docs/refs. 124/124.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ocument orgs Real-tenant check against build 176407869 (msazure/One) confirmed the ADO MCP can't reach that org (TF200016) but az can. So: - localization CONFIG carries the exact az reads (az_read: status via `az pipelines build show`; OneLocBuild@3 log id via `az devops invoke ... timeline`; log content via `... logs`). build()._trigger and the poller automation prompt now spell these out for msazure/One. - extract_pr() prefers the full PR URL the OneLocBuild@3 log prints after the id (real form includes /DefaultCollection/), falling back to the template; pr_url_template updated to the DefaultCollection form. - KNOWLEDGE explains MCP-can't-reach-msazure and the az read path. Also document the two-org reality in EXTERNAL-REFERENCES: Engineering lives in identitydivision (ADO MCP bound here); One lives in msazure.visualstudio.com (az only). 125/125. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ore EOD) When the completed localization run's PR is posted to the Code reviews chat, the message now @mentions the release engineer (release owner) and asks them to ensure the PR is merged before EOD. _review_post builds the Teams mention: an <at id="0"> tag in the html content matched by a mentions[] entry (id=owner UPN, userIdentityType=aadUser) that workiq_send_chat_message consumes. Falls back to a plain (unmentioned) post when no owner email is set. Tests + KNOWLEDGE updated. 126/126. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two per-step mock knobs (exercised in a real msazure/One test run, build
177042471):
- create_pr (input): overrides isCreatePrSelected on the trigger; set false to
run the pipeline WITHOUT creating a PR.
- send_to (post): redirects the completion PR post to your own chat ('me').
Applied by check-localization, since the post happens in the poll decider,
not build().
step_action's payload-override now only touches kind:payload knobs, leaving
input (build) and post (poller) knobs to their proper stages. 126/126.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… pipelines Fast-forwards Phases 0-1 offline, then runs the four build_verify steps LIVE (real az reads of the 2026-08 checker/orchestrator/ECS/Local MRWP runs) and halts at the go_test gate. Complements at_rc_gate (offline) and mid_build_verify_open (positions at entry). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…om real release The sim writes to .sim-runs and never touches the real release, so a normal `status` (which reads the real runs-root) won't reflect a sim run — by design. Add `sim status` to render the sandbox directly (auto-discovers the sim release), and make the SKILL/commands docs call out the isolation explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…L routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… it normally Per user feedback: the essential machinery is just a correct state.json (produced by the fast-forward, not hand-authored) + mocks. The isolated .sim-runs sandbox + sim status was unnecessary ceremony. Now: - sim run seeds the REAL runs-root by default (DEFAULT_SEED_RUNS), backing up any existing state to release-state.pre-seed-<ts>.json first. - Removed the sim status subcommand — a normal status reflects the seed now. - After a seed you use the ordinary skill commands (status/rc-report/next/approve). - --runs-root still lets you target a throwaway sandbox on purpose. - Docs (SKILL.md, commands.md, scenario headers) updated; +backup round-trip test. 169/169 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- build_verify now anchored CCD+1 — it opens the day after Code Complete, so the engineer wakes to a resume (the 8:00 phase-open push is delivered by the daily digest automation; the engine gates on the date). - New scout step rc_report (before go_test) emails the RC verification report to the release owner: pipeline health (checker/orchestrator + versions) and both MRWP runs' stage completion + top failing test suites, composed from LIVE data. Frames failures as bug-bash triage unless a stage never ran. Blocks if no owner email (set-owner). Scout composes; the skill sends via workiq_send_email. Sim marks it done w/o sending. - _common.py: rc_email/_rc_email_html/_rc_email_plain/rc_report_model formatters. - knowledge.yaml: build_verify.rc_report entry. Tests: rc_report email + block, updated phase-shape (6 steps, CCD+1) + gate-count + CCD+1 due timing. 170/170 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The phase-open 'resume' must land in the morning, not overnight. The daily digest automation ran 'every hour' (first tick could be 1 AM). Pinned to cron '0 8-18 * * *' so the first tick of each day is 08:00 local — a phase opening (e.g. Phase 2 on CCD+1) reaches the owner as an 8 AM resume — with hourly same-day catch-ups (idempotent + once/day de-dup) and nothing overnight. Updated the live 2026-08 automation and the provisioning doc so future releases provision it pinned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uites; drop dismissive line Addresses report-quality feedback: - The '(x2 more)' confusion was duplicate suite names: the test API returns the same suite as several runs. New pipelines.get_failed_tests aggregates by suite base name (strips the ' # <buildlabel>' run suffix), summing failures and deduping test names. - Lists the actual failing TEST names under each suite bullet (fetched from test/Runs/<id>/results?outcomes=Failed), bounded by max_result_calls/per_suite_cap. - release_report gains failed_suites per MRWP provider (with_failed_tests, default on). - Removed the 'No blocking issues - ... triaged in bug bash' line from both the CLI report and the email; the email drops the green triage box. Blocking-issues section still shows only when a stage never ran. - Both the rc-report CLI (_format) and the rc_report email render the new detail. - Tests: suite aggregation/dedup + failing-test-name rendering; updated rc-report test. 171/171 pass. Verified live against 2026-08 (real failing test names surfaced). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The autonomous worker shouldn't wait: 'every hour' (24x/day, incl. overnight) catches finished pipelines/steps as soon as they land. The 08:00-18:00 pin optimized for the digest email arriving in the morning but throttled the autonomous engine — wrong tradeoff. Reverts doc commit 9c9219f; the live automation was also reset to 'every hour'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… auto-detect at init
Two root causes made Phase 1 comms fire ~13h early in testing:
1) Due-ness used the HOST date (date.today()). On a UTC automation host the date
rolled at UTC-midnight (evening before, Pacific), opening phases early. Now the
engine evaluates due-ness + every fire_at_local on the OWNER's timezone.
- schedule.py: get_tz/now_local + today() is now tz-aware.
- engine.__init__ computes self.now_local + self.as_of from a tz/now; tz precedence
is arg -> state.timezone -> config/schedule.yaml -> DEFAULT_TZ (America/Los_Angeles).
2) The every-hour worker drained timed steps the instant a phase went due, ignoring
their fire_at_local. New Orchestrator._step_time_ready gates a timed step until its
wall-clock time (owner tz) on its fire day; wired into scout_pending, _step_sequential,
and _step_parallel. The dedicated cron automation calls step-action (build) directly,
so it still fires the step at the pinned time; missed steps catch up next day.
3) init now auto-detects the owner's IANA timezone (tzlocal) and PERSISTS it on the
release (state.timezone), so later headless runs — even in a UTC process — use the
owner's clock. --timezone overrides; config/schedule.yaml carries the default.
Deps: tzdata (IANA db on Windows) + tzlocal (detect). Recorded in requirements.yaml.
automations.fire_at() exposes a step's fire_at_local to the engine.
Tests: +5 (PT today, UTC-evening due-ness, timed-step gate + catch-up, state-tz
precedence, init capture). 176/176 pass. Code-reviewed clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…re %, visual dashboard
Report-quality improvements to the Phase-2 RC report (email, plain text, and CLI):
- Test-run classification: new pipelines.classify_test_run buckets each run into THREE
categories — unit (*_UnitTests), instrumented (*_InstrumentedTests), and UI automation
(everything else, incl. the (API NN) device suites and Lab Api Tests; "the rest are UI").
No "Other" bucket. get_test_summary now returns per-category {total,passed,failed};
get_failed_tests tags each failing suite with its category.
- The headline metric is now the UI-automation failure rate (the RC-critical bucket),
computed over UI tests only — not all tests. This also corrects an inflated count:
failed is total-passed-notApplicable per category, so skipped/NA tests no longer count
as failures (e.g. ECS UI 30 failed / 181, ~16.6% — not 104/5871).
- Visual dashboard (email): header banner, summary strip (UI failure rate / checker /
orchestrator), per-provider cards with a big UI-failure %, pass bars, a per-category
breakdown table (Unit/Instrumented/UI ← RC gate), and failing suites tagged by category
with per-suite % and test names.
- CLI rc-report + plain-text email show the same per-category breakdown.
- Tests: classifier (3 buckets, Lab Api → UI), suite aggregation w/ category, updated
rc_report email assertions (per-category headline). 177/177 pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cheduler is local, not UTC) Empirically verified 2026-08-20: a cron '37 9' fired at 09:37 PDT / 16:37 UTC. Docstring-only note so a UTC conversion is never added, which would shift every CCD-day comm by the host's UTC offset. No logic change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mentation and tests
…ent three-tier UI automation quality gate - Updated build_verify_live.yaml to auto-advance rc_report and position at bug-bash entry. - Modified mocks.local.example.yaml to reflect changes in RC report handling. - Enhanced rc_report.py to apply a three-tier UI gate, determining pass/warn/attention outcomes. - Adjusted sim.py to change auto-approve gates from go_test to bash_done. - Revised SKILL.md and commands.md to update user guidance on RC verification and command usage. - Updated build_verify.md to clarify the removal of the go_test gate and the new RC report process. - Enhanced _common.py to summarize UI failing suites for better reporting. - Updated tests to reflect the removal of the go_test gate and ensure correct behavior in the new flow.
Design-audit follow-up to the go_test removal — eliminate the same 'step id hardcoded across modules' smell in two more places: 1) automations planner no longer special-cases ccd.localization. A step module may now declare automation_prompt(release, spec) (single source of truth, like fire_at_local); localization's bespoke trigger/poller prompts move into steps/ccd/localization.py and _prompt_for() delegates generically. 2) render/digest no longer hardcode the build_verify phase id for the RC pipeline-run line — driven by a data flag (show_pipeline_runs) in phases.yaml, propagated via engine _active_phase_report. Engine core confirmed free of hardcoded step ids. 181/181 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e in the report
rc_report + the 90% UI gate no longer re-discover pipeline ids live — they read the RECORD the verification steps store in state.pipeline_runs.
Schema: { checker{run_id,when}, orchestrator{run_id,versions{},parked}, rcs:[{rc,ecs{...},local{...}}] } — 1 checker + 1 orchestrator, N RC iterations (a re-trigger appends a new ecs/local pair), latest = rcs[-1]. Each provider slot snapshots stage completion + the Test-tab summary + failing suites, so the gate is deterministic/replayable from state alone.
Legacy flat pipeline_runs auto-migrates to the nested shape on ReleaseState.load. The live rc-report diagnostic still reads fresh and refreshes the record. render._pipelines_line reads the nested shape (latest RC). 183 tests pass (incl. migration + append-new-rc coverage).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ered = passed + warn) Unit tests run under a retry rule: a flaky test can appear several times in one run (e.g. Passed/Failed/Passed) and ADO's run aggregate still counts it as a failure. Reconcile per-result by testCaseTitle so a test that passed on any attempt counts as PASSED; a failed-then-passed test is RECOVERED (counted passed, surfaced as a warning); only never-passed tests are real failures. Unit-only — UI/instrumented (and the UI gate) are unchanged. reconcile_retries() + a paged per-run results fetch; get_test_summary re-reads failing UNIT runs per-result; get_failed_tests drops recovered unit titles (and fully-recovered suites); the RC email (plain+HTML) and rc-report diagnostic show a retry warning listing recovered unit tests. Also fixes retry double-counting in totals. Verified live on build 1681651: testNullDrsMetadata + gated_notEnabled_logsNothingAtAll reconcile to passed, unit failed=0. 187/187 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-model builder, test net-guard H2 (single-source coordinates): the release toolchain spans MULTIPLE ADO orgs (identitydivision/Engineering for the verification chain, msazure/One for localization+CG, identitydivision/IdentityWiki for the wiki). tools/pipelines.py now names them explicitly (IDENTITYDIVISION/MSAZURE hosts, ENGINEERING_ORG/ENGINEERING_PROJECT) instead of a misleading global ORG/PROJECT; build_verify (Engineering-only) aliases them, cron/wiki import the right target, localization/cg keep their own. H1 (one RC-model builder): assemble_rc_model() is the single canonical builder; release_report (live) and rc_report_model (state) both route through it so they can't drift on shape or problem messages. Guard test asserts agreement; live-verified on 2026-08. L2 (test net-guard): tests/conftest.py autouse fixture makes any un-mocked ADO/az call raise loudly instead of hanging. 188 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…derers, tidy imports/excepts M1: extract the ~250-line status view-model (_active_phase_report/_phase_map/_current_steps/_hold_view/_scheduled_view/status_report) into orchestrator/status_views.py as StatusViewMixin; engine.py 783->540 lines. Behaviour identical. M2: sort_failed_suites() + shared recovered_unit_tests() reused across the three RC renderers (plain email / HTML email / CLI report) — one place to change ordering. M3: format_versions() helper collapses six duplicated 'Common/Msal/Broker' join idioms. M4: hoisted the safe lazy imports in build_verify/_common (tools.pipelines, state, datetime, outcomes, mockctx) — no import cycle; left the automations/steps lazies that guard real cycles. L1: narrowed parse-y broad excepts (discovery JSON, engine/schedule YAML+zoneinfo, cli stream reconfigure) to specific exception types; left the intentional best-effort IO ones. L3: documented the orchestrator healthy=True invariant in rc_report_model. 188 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…c, not 'Pending' UX fix: the four Phase-2 verification steps are agent steps Scout runs itself in-process, but they rendered as 'Pending' (⬜) next to the scout rc_report step's 'Scout runs this — automatic' (🤖) — a confusing mixed message implying the human was waiting on the agent steps. New 'auto' display state for engine-run agent steps renders IDENTICALLY to the skill-run 'scout' state (🤖 'Scout runs this — automatic'). 'Pending' is now reserved for a human step queued behind a dependency, so it always means 'you'll act here'. Everything needing the user stays distinct (⏸ gate, 📌 reminder, ⛔ blocked). Render-only; no state-machine change. 189 tests pass (added a display-contract regression test). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…alse 2nd active phase) Bug: _phase_map marked ANY phase with done>0 as 'current' (⏸). After an upstream reopen left stale progress in a later phase, TWO phases rendered as 'in progress' — e.g. Phase 2 (frontier, 4/5) AND Phase 3 (2/5 leftover). Confusing: looked like Phase 3 ran while Phase 2 was incomplete. Fix: derive the frontier (first-incomplete included phase) once and mark ONLY it 'current'; a later phase with stale partial progress renders 'pending' (its count still shows). Ordering preserved so a not-yet-due frontier stays 'scheduled'. current_phase_obj/current_step_name are now frontier-derived too (equal to the engine cursor during real holds — no behaviour change). Display-only; the engine already processed the frontier first and never re-ran downstream. 190 tests pass (added a regression reproducing the two-active-phase state). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…in-progress runs) An MRWP RC run that is still notStarted/inProgress no longer blocks as an aborted pipeline. verify_mrwp now checks the run's overall status first (new pipelines.get_build_status) and returns a new InProgress outcome when it isn't completed - a pending stage on a live run means "not run YET", not aborted. Engine records this as a new non-blocking StepState status "in_flight": not added to pending_human, release stays running (no user action), drain returns a "waiting" hold, and in_flight_since is stamped for the upcoming 6h poller nudge. The step stays not-done so it re-runs on every next/poll until the run completes, then the normal gate applies. Display: in_flight renders as "RC running - Scout is polling" in CLI status + email digest. Adds 2 tests; suite 192 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…est RC) When the human explicitly signals a NEW RC was triggered - a flaky-suite re-run, or the orchestrator re-running RC testing after a broker cherry-pick - `rc-retriggered --release <id> [--reason]` reopens the two MRWP verifies + rc_report (checker_fired / orchestrator_health are left intact; a re-triggered RC re-runs MRWP against the same orchestrator run). mrwp_run_ids already picks the highest (newest) run id, so the engine re-resolves the new RC and re-applies the gate on the next tick/poll; the status-aware verify holds while it is still in-flight, so an early poll cannot false-fail an in-progress RC. Clears the reopened steps from pending_human, flips status back to running, and journals an `rc_retriggered` event. Adds a test; suite 193 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tomation) Adds the poller that watches an in-flight re-triggered RC and drives it to a verdict: * poll-rc command: advances the drain (in-flight verify re-checks the run's live status), then emits a deterministic decision - waiting / nudge / resolved / blocked / idle. A run in-flight past NUDGE_AFTER_HOURS (6h) yields ONE courtesy heads-up to the owner (email + Teams text), stamped nudged_at so it never repeats. Not a failure - Scout keeps polling. * build-verify-rc-poller automation (config/automations.yaml): a 30-min interval poller driving build_verify.rc_report, marked on_demand so it is provisioned ONLY when an RC is re-triggered (not at release start) and torn down when poll-rc reports resolved. plan() now surfaces on_demand. * rc_report.automation_prompt: the bespoke poll instruction (act on the decision; deregister on resolved), owned by the step like localization's. Adds 3 tests (waiting->nudge->dedupe, resolved/blocked/idle, plan shape); suite 196 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The rc_report `attention` block now spells out THREE exits instead of a vague "fix or override", matching the agreed Phase-2 blocked-state handling: 1. Re-trigger (flaky) - re-run the failed RC test run, then `rc-retriggered --release <id>`; Scout tracks the newest RC (holds in-flight, polls 30m, re-gates on completion). 2. Cherry-pick (real bug) - patch via the broker cherry-pick process (link surfaced), then `rc-retriggered` so Scout tracks the fresh RC to completion. 3. Override (LAST RESORT) - `skip ... --reason`, framed explicitly as a team decision to be discussed first (proceeding to Bug Bash with this many UI failures is not a default). Adds CHERRY_PICK_TSG. Updates the skill docs to present all three (never collapse to two): reference/phases/build_verify.md (3-exit flow + in-flight vs blocked), SKILL.md block handling exception, reference/commands.md (rc-retriggered + poll-rc). Strengthens the gate test; suite 196 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Relocate release-owned automations into their release folder and remove dead legacy
scaffolding (this project is pre-production — no old on-disk state can exist):
Registry ownership:
* Release-scoped automations now live in <runs_root>/<release>/_automations.json,
co-located with that release's release-state.json so ownership is explicit and they're
removed with the release folder at close. The machine-wide <runs_root>/_automations.json
now holds ONLY shared automations. AutomationRegistry takes an optional release; list/
register/deregister read+write the right file(s). Relocated the existing dev registry.
Legacy removals (can't exist without production data):
* state.migrate_pipeline_runs (flat->nested) + _versions_str_to_dict helper + its test —
the flat pipeline_runs shape only briefly existed earlier in dev before the nested
redesign; callers now read state.pipeline_runs directly.
* ReleaseState.last_notified ("kept for load compat") — read nowhere.
* automation sync's steps-fallback for entries "predating slug" — register always sets
slug now, so match by slug only.
Kept: ReleaseState.load unknown-key drop (genuine robustness for the unattended
automation, not legacy). Docs updated. Suite 196 green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ator) Replaces the single clone_plans stub with two real ADO-backed agent steps, per the distinct team procedures: * clone_plans_broker — COPIES the Broker master test plan (#2007357, area Engineering\Auth Client\Broker\Android) to a new plan "Android Monthly Release - <Mon YYYY>", referencing existing test cases (ADO clone default). Follows the eng.ms broker test-plans doc. * clone_plans_auth — CREATES a query-based (dynamic) test suite under "MSAuthenticator Test Passes" (#714514/714515) named "Android/release/MM/YYYY", WIQL mirroring the live suite "Android/release/08/2024" (tag Android, Identity Apps area, not Closed/IgnoreOnPrem). Follows the IDWiki 33580 doc and STOPS before assigning testers (later step). Both are idempotent: the created plan/suite id is stashed on the step and re-confirmed on re-run (auth also reuses a same-named suite) so a re-entered Phase 3 never makes duplicates. New tools/testplans.py wraps the ADO test-plan clone + dynamic-suite-create APIs; adds a pipelines._ado_rest_send POST primitive (the first ADO write path). Query/name shapes and plan ids were verified live read-only against the real plans. config/phases.yaml splits the step; knowledge.yaml documents both for step-info; tests cover clone/idempotency/block offline via mock knobs (net-guarded). Suite 202 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…verride knob Live-tested both steps against real ADO (Engineering) with TEST-prefixed names, then deleted the artifacts. The live run exposed two real bugs, now fixed: 1. Broker clone HTTP 400 "CopyAncestorHierarchy cannot be false if multiple suite ids are provided" — the clone body set copyAncestorHierarchy=false while passing the source root suite id. Set it to true; the clone now succeeds (verified: cloned plan + suites, referencing existing test cases). 2. clone_plans_auth created a DUPLICATE suite because its name-scan only saw the first page — ADO returns the suites-list continuation token in the `x-ms-continuationtoken` RESPONSE HEADER, not the body. Added pipelines._ado_rest_get_h (header-aware GET) + _ado_rest_get_all (follows the header token); find_child_suite_by_name now pages all suites and correctly finds an existing same-named suite (verified live — no duplicate). Adds a `name` mock knob to both steps to override the derived plan/suite name (enables safe 'TEST ...' live runs and staging). Extends the test net-guard to _ado_rest_get_h / _ado_rest_send. New offline tests: name override + header-continuation paging. Suite 204. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t IgnoreOnPrem")
The Authenticator bug-bash query was over-including: it filtered Android test cases by
"not IgnoreOnPrem" (copied from the stale 2024 suite), yielding 66 cases vs prod's 45.
Live diff vs the current prod suite 3728419 ("Android release/08/13/2026") showed prod is
a strict subset — the 21 extras all LACK the 'ReleaseBugBash' tag (they're ComposeTesting
/ ComposeSettingTesting / DarkMode cases, not the curated bug-bash set). The IDWiki doc
and prod both filter on [System.Tags] contains 'ReleaseBugBash'. Because ADO's tag
`contains` is a substring match, that single clause also captures month-specific
'ReleaseBugBash<Month>' tags — exactly as the doc describes.
Corrected auth_bugbash_query() to filter on 'ReleaseBugBash'; verified live the new query
yields 45, matching prod. Updates the test assertion, module docstring, and knowledge FAQ
(adds a which-cases entry). Suite 204 green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…D (match prod) The Authenticator suite name now follows the prod/IDWiki convention exactly - "Android release/MM/DD/YYYY" with the day taken from the release CCD (e.g. CCD 2026-08-13 -> "Android release/08/13/2026", matching prod suite 3728419). Was the day-less, extra-slash "Android/release/08/2026". auth_suite_name(ccd) now derives from the CCD date; the step passes state.ccd and blocks with a clear message if no CCD is set (the name needs the day). Updates the docstrings, knowledge FAQ, and tests; adds a no-CCD block test. Verified live (name + suite create). Suite 205 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ms empty") Investigating a broker clone that "seemed empty" revealed the real gap: ADO's CloneOperation is ASYNCHRONOUS. clone_broker_plan POSTed the clone and returned the destination plan id straight from the POST response, before the background suite/test-case copy finished — so a downstream read (or the user opening the plan) right after the step reported done could see an empty/partial plan. Verified live: a completed clone is a perfect copy of the master (45 suites: 16 static + 29 dynamic, 276 test cases), and cloneStatistics shows clonedTestCasesCount=0 / totalTestCasesCount=271 — i.e. it correctly REFERENCES existing test cases, not duplicates. The clone status lives at cloneOperationResponse.state. clone_broker_plan now polls the CloneOperation until state 'succeeded' (blocks on 'failed') before returning, so the plan is fully populated when the step reports done (measured ~14s for the master). Adds a _clone_op helper for the nested response shape and 2 tests (async-poll-to-succeeded, block-on-failed). Suite 207 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.