test(firefox): fixme flaky codegen pierceFrames disambiguation on Intel macOS - #42131
Open
microsoft-playwright-automation[bot] wants to merge 2 commits into
Open
test(firefox): fixme flaky codegen pierceFrames disambiguation on Intel macOS#42131microsoft-playwright-automation[bot] wants to merge 2 commits into
microsoft-playwright-automation[bot] wants to merge 2 commits into
Conversation
…el macOS
Hi, I'm the Playwright bot. `cli codegen with pierceFrames › should prepend the frame selector to disambiguate` is flaky, but **only on `firefox-macos-15-large` (Intel macOS)** — 8 failures / 46 runs (17.4%) in the recent window, and zero failures on every other bot (chromium, webkit, firefox arm64, ubuntu). Reproduced locally on Intel macOS with `--repeat-each=15 --workers=6`: 3 flaky, all the same failure.
Root cause is a timing race, not a bad assertion: `buildFullSelectorForFrame` races the cross-frame piercing-selector resolution against a hardcoded 2000ms deadline and, on timeout, falls back to the `internal:control=no-pierce-frames` opt-out. Under load on the Intel macOS runner that race times out, so the recorder emits `page.pierceFrames({ pierce: false }).locator('#frame1')…` instead of the expected piercing selector. `test.slow()` wouldn't help — the 2s budget is internal to the recorder, independent of the test timeout. Same Intel-macOS-firefox-under-load class as #41811.
Scoped `test.fixme` to `browserName === 'firefox' && isMac && process.arch === 'x64'`; the test still runs everywhere else.
Run history for `firefox-macos-15-large` (oldest first, 46 runs):
[🟧](https://github.com/microsoft/playwright/actions/runs/30490403874/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30493031010/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30495285330/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30530696555/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30534756293/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30539645601/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30542293233/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30542345092/attempts/1)[🟧](https://github.com/microsoft/playwright/actions/runs/30559579795/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30565052100/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30568188818/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30568293053/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30568320872/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30577572373/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30581383958/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30581357288/attempts/1)[🟧](https://github.com/microsoft/playwright/actions/runs/30581405526/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30585190825/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30618799783/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30624512498/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30653773411/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30667286182/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30667347031/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30833251658/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30850656923/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30850749658/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30854798001/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30859315578/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30859305376/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30859360344/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885108613/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885140825/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885208556/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30885237385/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30885274230/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30917613834/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30927071352/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30927719515/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30930057055/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30933859878/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30934926278/attempts/1)[🟥](https://github.com/microsoft/playwright/actions/runs/30940141663/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30945227770/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30954253724/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30954381228/attempts/1)[🟩](https://github.com/microsoft/playwright/actions/runs/30955628029/attempts/1)
<details>
<summary>What I verified locally (Intel macOS, x64)</summary>
- Reproduced: `npm run ftest -- tests/library/inspector/cli-codegen-3.spec.ts:1169 --repeat-each=15 --workers=6` -> 3 flaky with the `pierce: false` opt-out.
- After fix: firefox -> `1 skipped`; chromium (`npm run ctest -- …:1169`) -> `1 passed`.
- `npm run flint` -> clean.
- CI run: https://github.com/microsoft/playwright/actions/runs/30984018624
</details>
Suggested-reviewer: dgozman
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Test results for "tests 1"12 flaky51217 passed, 1190 skipped Merge workflow run. |
Contributor
Test results for "MCP"8050 passed, 1284 skipped Merge workflow run. |
Member
|
@copilot if the 2s race is the problem, maybe we can make those 2s configurable and extend them just under intel macOS tests? It seems like that would be the better fix. |
Co-authored-by: Skn0tt <14912729+Skn0tt@users.noreply.github.com>
Contributor
Implemented in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, I'm the Playwright bot.
cli codegen with pierceFrames › should prepend the frame selector to disambiguateis flaky, but only onfirefox-macos-15-large(Intel macOS) — 8 failures / 46 runs (17.4%) in the recent window, and zero failures on every other bot (chromium, webkit, firefox arm64, ubuntu). Reproduced locally on Intel macOS with--repeat-each=15 --workers=6: 3 flaky, all the same failure.Root cause is a timing race, not a bad assertion:
buildFullSelectorForFrameraces the cross-frame piercing-selector resolution against a hardcoded 2000ms deadline and, on timeout, falls back to theinternal:control=no-pierce-framesopt-out. Under load on the Intel macOS runner that race times out, so the recorder emitspage.pierceFrames({ pierce: false }).locator('#frame1')…instead of the expected piercing selector.test.slow()wouldn't help — the 2s budget is internal to the recorder, independent of the test timeout. Same Intel-macOS-firefox-under-load class as #41811.Scoped
test.fixmetobrowserName === 'firefox' && isMac && process.arch === 'x64'; the test still runs everywhere else.Run history for
firefox-macos-15-large(oldest first, 46 runs):🟧🟥🟩🟩🟩🟩🟩🟩🟧🟩🟩🟩🟩🟩🟥🟩🟧🟩🟩🟩🟩🟩🟩🟩🟥🟩🟩🟩🟩🟥🟩🟩🟩🟩🟥🟩🟩🟩🟩🟥🟥🟥🟩🟩🟩🟩
What I verified locally (Intel macOS, x64)
npm run ftest -- tests/library/inspector/cli-codegen-3.spec.ts:1169 --repeat-each=15 --workers=6-> 3 flaky with thepierce: falseopt-out.1 skipped; chromium (npm run ctest -- …:1169) ->1 passed.npm run flint-> clean.Suggested-reviewer: dgozman
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com