test: give the jobs manager a shutdown budget that survives a loaded machine

[packetloss404]

Chases down the TestResubmit_IsOnlyAllowedOnce failure left open after the sharing-violation fix.

What it actually was

It failed rarely on Windows, at ~17s, with an error that named nothing:

TempDir RemoveAll cleanup: unlinkat ...: The directory is not empty.

No assertion in that test was involved. It never reached one.

newTestManager gave Shutdown a hard-coded, unscaled two seconds and discarded the result. Shutdown returns when its timeout expires whether or not the workers stopped — and says which in the error it returns:

case <-time.After(timeout):
    shutdownErr := fmt.Errorf("jobs.Shutdown: %d workers still running after %s", ...)

Under a parallel suite two seconds expired routinely. The cleanup returned having shut nothing down; the workers kept writing job records into the t.TempDir() Go was about to remove; and on Windows a directory that is gaining files cannot be removed. So the test died in cleanup, long after the code under test had behaved correctly.

TestResubmit_IsOnlyAllowedOnce was the likeliest victim because it is the only Resubmit test that spawns a job and never waits for it — its worker is always still running at cleanup.

Fix

• The budget scales through internal/testwait, like every other deadline in the suite: a busy machine gets slack, a genuinely stuck worker still fails.
• The error is reported instead of discarded. "N workers still running after 20s" is the one sentence that identifies this immediately; throwing it away is what made a shutdown that shut nothing down look like success.
• The same unscaled literal appeared at 14 other call sites in internal/jobs and internal/app. Scaled too, so the flake relocates to none of them.

Evidence

	result
Forced (budget cut to 1ns)	reproduces 6/6 with exactly that message
Natural (4 concurrent full-suite runs)	reproduced on attempt 1, at 19.26s
After the fix, same load	500 iterations clean

• go test ./... clean; go vet and golangci-lint (repo config, truncation disabled) under GOOS=linux, darwin, windows: 0 non-gofmt findings. (gofmt reports every file locally because the working tree is CRLF; that is the known local-only artefact.)

Separately: an unrelated flake found while verifying

TestStallGuard_TickKeepsAlive in internal/provider is independently flaky — same class (an unscaled 30ms real-time deadline, ticked every 10ms, which load starves), different package. Reproduced 3 failures in one batch of 25 on a clean tree with none of these changes, so it is not caused by this work and is not fixed here. It needs a different remedy than testwait (whose 5s floor would make that test take ~13s), so it is left for a deliberate change rather than folded in.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-05T19:31:08.763601Z	afd5367	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.