OCE report: add Authenticator app telemetry and turn the skill into a router, Fixes AB#3731627 - #461
Open
Shahzaib (shahzaibj) wants to merge 1 commit into
Conversation
… router One slash command now produces both weekly reports. Previously an on-call engineer had Broker telemetry only; Authenticator app health lived in a Kusto dashboard nobody opened during a rotation. SKILL.md becomes a thin router: it resolves the reporting window ONCE, picks a mode, and dispatches. All Broker analysis moves verbatim into assets/playbooks/broker.md; the new assets/playbooks/authapp.md is its Authenticator counterpart. The two playbooks are deliberately never read into one context -- in `both` mode they run as parallel sub-agents, because their Kusto conventions are mutually incompatible (Broker must never sum(countDevices) and uses HLL; Authenticator's sum(SucceededDCount) is the correct idiom there). Interleaving them risks writing one app's numbers with the other's rules. Modes: `both` (default) | `broker` | `authapp`. Outputs land in %USERPROFILE%\android-oce-reports\ as two separate reports plus, in both mode, an oce-index-<curEnd>.html digest built by the new build-index.ps1. Two reports rather than one merged document: the apps have different owners, different triage ladders and different escalation paths, and a merged report forces every reader through the half they do not own. Authenticator coverage: scenario funnels (Passkey / Entra MFA / Entra PSI / MSA NGC+SA), error-reason decomposition, abandonment, Broker API responsiveness, version share, and an optional App Center crash layer (--skip-crashes). bootstrap-report.ps1, run-kql.ps1, validate-report.ps1 and find-suspect-prs.ps1 gain -App broker|authapp (validate-report.ps1 also gains the Authenticator check profile, filling the reserved check 12 slot); find-suspect-prs.ps1 gains -Repos so it can scan the authenticator repo instead of broker/common. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
✅ Work item link check complete. Description contains link AB#3731627 to an Azure Boards work item. |
|
❌ Invalid work item number: AB#3731627 Click here to learn more. |
|
✅ Work item link check complete. Description contains link AB#3731627 to an Azure Boards work item. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On-call engineers had Broker telemetry in a weekly report and Authenticator app health in a Kusto dashboard nobody opened during a rotation. The ask was explicit: one slash command, both reports — not two commands, and not one merged document.
What this does
SKILL.mdbecomes a thin router. It resolves the reporting window once, picks a mode, and dispatches. All Broker analysis moves verbatim intoassets/playbooks/broker.md; the newassets/playbooks/authapp.mdis its Authenticator counterpart.both(default)oce-index-<curEnd>.htmlbrokeroncall-wow-report-<curEnd>.htmlauthappauthapp-wow-report-<curEnd>.htmlTwo reports, not one
The two apps have different owners, different triage ladders and different escalation paths. A merged report forces every reader through the half they don't own. The
bothmode instead emits two standalone reports plus a one-page index digest linking them.The playbooks are never read into one context
In
bothmode they run as parallel sub-agents. This isn't only about wall-clock — their Kusto conventions are mutually incompatible:sum(countDevices)is actively wrong.sum(SucceededDCount)is the correct idiom.Interleaving them in one context risks writing one app's numbers under the other app's rules. The router says so explicitly, and the shared hard-rules section calls out that app-specific rules are never interchangeable.
Authenticator coverage
Scenario funnels (Passkey / Entra MFA / Entra PSI / MSA NGC+SA), error-reason decomposition, abandonment, Broker API responsiveness, version share, and an optional App Center crash layer (
--skip-crashes, since it needs a secret).Plumbing
bootstrap-report.ps1,run-kql.ps1,validate-report.ps1andfind-suspect-prs.ps1all gain-App broker|authapp.validate-report.ps1also gains the Authenticator check profile — which fills the check 12 slot deliberately reserved in the previous PR, so this validator diff is a pure insertion with no renumbering.find-suspect-prs.ps1gains-Reposso it can scan the authenticator repo instead of broker/common.New
build-index.ps1reads the headline KPI tiles out of both finished reports and emits the digest. It is a digest, not an analysis — a cross-app finding gets written into both reports, and the index just links them.Scope
This PR is purely additive Authenticator support plus the router refactor. It contains no Broker behaviour changes — those are in the parent PR, and the window-alignment fix is in the child. That separation is the whole point of the split.
SKILL.mdshrinks substantially because its Broker content moves toassets/playbooks/broker.mdrather than being deleted.Verification
Full E2E run in default
bothmode: both reports generated, both validators pass, index built. Reports land in%USERPROFILE%\android-oce-reports\— outside the workspace, so they can't be committed by accident.Stack
This is 2 of 3:
masterFixes AB#3731627