Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions .github/workflows/ci-and-labels.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions plans/149-nightly-full-viewport-e2e-2026-09-24.md
Original file line number Diff line number Diff line change
@@ -1,72 +1,81 @@
# Plan 149 — Nightly E2E Sweep Covers Every Viewport (2026-09-24)

**Type**: CI coverage extension + test robustness
**Scope**: `.github/workflows/ci-and-labels.yml`, `e2e/semantic-search.spec.ts`
**Follows**: plans/148 §6.1–6.2, plans/122 W2 (which introduced the nightly E2E job)

## 1. Problem

CI's E2E job installs **Chromium only** and runs `pnpm run test:e2e
--project=chromium`. The `mobile` (iPhone 13, 390×664) and `tablet` (iPad Pro 11,
834×1194) projects are **WebKit** device descriptors, so they have never run in
CI — not on PRs, not in the nightly sweep that plans/122 added.

That gap is not theoretical: the graph label-click defect in plans/148 reached
`main` and was only caught by a manual four-project sweep. Playwright's own log
showed the failure at `chromium`, but the sweep is also the only way
viewport-specific layout regressions surface at all.

### The nightly never ran at all

Checking the most recent scheduled run (`2026-09-24T07:59Z`, head `064702a`)
before changing anything showed:

```
Detect Changes: completed/success
Quality Gate: completed/skipped
Unit Tests: completed/skipped
E2E Tests: completed/skipped
Build: completed/skipped
```

`e2e-tests` declares `needs: [changes, unit-tests]`, and `unit-tests` excludes
scheduled events (`github.event_name != 'schedule'`). GitHub skips a job whose
needed job was skipped unless the dependent job uses a status function, and the
E2E condition has none — so the "nightly E2E backfill" added by plans/122 W2 has
been a no-op since it landed. Nothing in the workflow said so: the job that would
have reported it was the one being skipped.

Two things therefore stood in the way of a working nightly:

1. **WebKit is not installed** in that job (`playwright install --with-deps
chromium`), and installing it adds apt dependencies — fine nightly, wasteful
per PR.
2. **`e2e/semantic-search.spec.ts` is load-sensitive.** It waits up to 20 s for
transformers.js to finish failing its blocked CDN fetches before the lexical
fallback hint appears. Under a four-project sweep it failed on both the
attempt *and* the retry, while passing 9/9 in isolation. A nightly that goes
red for that reason is worse than no nightly.

## 2. Change

**Workflow** (`e2e-tests` job):

| Event | Browsers installed | Projects run |
|---|---|---|
| `pull_request` | chromium | `--project=chromium` |
| push to `main` | chromium | `--project=chromium` |
| `schedule` (nightly, main) | chromium + webkit | all four |
| `workflow_dispatch` | chromium + webkit | all four |

`timeout-minutes` goes 20 → 40: PR runs still finish in ~3 minutes, and the
nightly now runs 596 tests across four projects on WebKit as well as Chromium.
The existing `actions/cache` step for `~/.cache/ms-playwright` keeps the WebKit
download off the nightly's critical path after the first run.

The condition is written as `event_name == 'schedule' || event_name ==
'workflow_dispatch'` rather than `event_name != 'pull_request'`. The first
version used the negation, which silently included **pushes to `main`** — every
frontend merge then paid the four-project cost (~10 minutes, observed on the
merge that landed this work). A push to `main` is a merge the PR already
validated; the sweep belongs to the nightly that exists for that gap. The
contract test now pins the sweep to those two events.

**Workflow** (`unit-tests` job): the `github.event_name != 'schedule'` exclusion
is dropped, so the job runs nightly. This is the fix for the silently skipped
sweep — `e2e-tests` needs it, and a skipped dependency skips the dependent. The
alternative (keeping the exclusion and giving `e2e-tests` an `always()`/`!cancelled()`
escape) relies on status-function semantics that cannot be exercised by a manual

Check notice on line 78 in plans/149-nightly-full-viewport-e2e-2026-09-24.md

View check run for this annotation

nexus-check / GitNexus

Changed symbol: 2. Change

`2. Change` (Section) is directly changed by this PR. PR-wide downstream impact: 0 direct dependent(s), 0 indirect. See the check summary for the impacted-file breakdown.
dispatch, whereas "the dependency runs in every event that reaches it" is
provable from the run history and testable in the workflow contract test. The
nightly therefore also runs the unit suite on `main`, which is a bonus signal
Expand Down
44 changes: 26 additions & 18 deletions src/lib/__tests__/workflows.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -149,24 +149,32 @@
run: string
}

// Both blocks are `if pull_request … else …`, so splitting on `else` yields
// the PR branch and the nightly/dispatch branch. Asserting the exact
// commands per branch is the point: `--project=chromium` is a prefix of the
// nightly command and `chromium` is a prefix of `chromium webkit`, so
// substring checks would still pass on a Chromium-only nightly.
const [prInstall, nightlyInstall] = install.run.split('else')
expect(prInstall).toContain('pnpm exec playwright install --with-deps chromium')
expect(prInstall).not.toContain('webkit')
expect(nightlyInstall).toContain('pnpm exec playwright install --with-deps chromium webkit')

const [prRun, nightlyRun] = runTests.run.split('else')
expect(prRun).toContain('pnpm run test:e2e --project=chromium')
expect(nightlyRun).toContain('pnpm run test:e2e')
expect(nightlyRun).not.toContain('--project')

// Both branches must be conditioned on the event, not merely present.
expect(install.run).toContain('github.event_name')
expect(runTests.run).toContain('github.event_name')
// Splitting on `else` yields the sweep branch and the default branch.
// `condition` is everything before `then` (only the first segment has one);
// `command` is the first line that is neither the condition nor `fi`.
const condition = (block: string): string => block.split('then')[0].trim()
const command = (block: string): string =>
block
.split('\n')
.map((line) => line.trim())
.find((line) => line.length > 0 && !line.startsWith('if [') && line !== 'fi') ?? ''

Check warning on line 160 in src/lib/__tests__/workflows.test.ts

View check run for this annotation

nexus-check / GitNexus

Assert the install block's event gate as well as the test-run block's

The new assertions only prove that the install branches contain the two commands (lines 157–160); unlike the run block, they never assert which events select the WebKit branch. Consequently, changing the install condition back to an `event_name != pull_request` check would still satisfy this test while causing every push to main to install WebKit. That violates the documented event table in plans/149-nightly-full-viewport-e2e-2026-09-24.md lines 54–72, which specifies Chromium-only browser installation on pushes and says the contract test pins the sweep to schedule/dispatch.

// The exact condition matters, in both blocks: substring checks let a
// sweep that also runs on pushes (`… || event_name == "push"`) pass, and
// a Chromium-only sweep passes `toContain('pnpm run test:e2e')` because
// `--project=chromium` is a prefix of it (plans/149).
const SWEEP_CONDITION =
'if [ "${{ github.event_name }}" = "schedule" ] || [ "${{ github.event_name }}" = "workflow_dispatch" ];'

const [sweepInstall, defaultInstall] = install.run.split('else')
Comment thread
d-oit marked this conversation as resolved.
expect(condition(sweepInstall)).toBe(SWEEP_CONDITION)
expect(command(sweepInstall)).toBe('pnpm exec playwright install --with-deps chromium webkit')
expect(command(defaultInstall)).toBe('pnpm exec playwright install --with-deps chromium')

Check warning on line 172 in src/lib/__tests__/workflows.test.ts

View check run for this annotation

nexus-check / GitNexus

The event assertions do not exclude pushes from the sweep

The comment says the sweep is limited to scheduled and manual events, but lines 170–172 only require that both event names occur and that `pull_request` does not. A predicate such as `event_name == "schedule" || event_name == "workflow_dispatch" || event_name == "push"` passes all three assertions while selecting the full E2E suite on pushes. The documented contract explicitly assigns pushes to the Chromium-only/default branch (plans/149-nightly-full-viewport-e2e-2026-09-24.md lines 54–72).

const [sweepRun, defaultRun] = runTests.run.split('else')
expect(condition(sweepRun)).toBe(SWEEP_CONDITION)
expect(command(sweepRun)).toBe('pnpm run test:e2e')
expect(command(defaultRun)).toBe('pnpm run test:e2e --project=chromium')
})

it('should not let a skipped dependency silence the nightly E2E sweep', () => {
Expand Down
Loading