🐛 bug(contribute): a 10–30 min contributor outage double-assigns its issue — the release cooldown (10m) lapses before the resume lease (30m) expires

[Danathar]

A contributor whose connection drops for between 10 and 30 minutes can have its in-flight issue handed to a second contributor while it is still able to resume — both then work the same issue and open duplicate PRs.

Found by code audit of v4 at ff0b345 (v4.67.0) and reproduced with a Go test against the real hub over local sockets. Treat the production impact as the consequence the code implies.

What happens

Two independent clocks are supposed to keep one issue with one contributor:

• The release cooldown. When a socket drops, releaseOnDisconnect clears the in-flight task and books a short cooldown so the issue is not instantly re-offered during the reconnect window. That cooldown is failedTaskCooldownMinutes = 10 minutes (contribute_cooldowns.go:19, booked by bookReleaseCooldown at contribute_ws.go:1870).
• The lease. The disconnect deliberately keeps the server-issued lease so the returning relay can resume its task. The lease stays re-adoptable for leaseTTL = wsTaskTimeout = 30 minutes, measured from the last progress report (contribute_leases.go:64).

The two windows are different lengths, and nothing else covers the gap between them. selectTask excludes an issue from re-offer only if a live connection currently holds it (contribute_select.go:337) or if it is in cooldown; a lease is not consulted as a hold outside the brief 2-minute post-restart grace. So between 10 and 30 minutes after the last progress report from a disconnected relay:

• the relay is gone, so the issue is not in the live in-flight set,
• the 10-minute release cooldown has lapsed, so the issue is not excluded by cooldown,
• but the 30-minute lease is still alive, so the moment the relay reconnects it re-adopts the task.

If a second contributor asks for work during that window, selectTask offers them the same issue. When the original relay reconnects (it keeps its task locally and re-asserts it on auth_ok, bin/contributor-relay.js:4622), lookupLease matches and it resumes — resumeTaskToken even mints it a fresh push credential. Now two contributors hold the same issue with valid tokens.

When it bites

The design note for #5322 reasons that the release cooldown "comfortably outlasts the reconnect backoff," which is true for a normal flap: the relay reconnects within its 1–60s backoff, well inside the 10-minute cooldown, and repopulates the live in-flight set before it lapses. The gap opens for a medium-length outage — 10 to 30 minutes — that the relay survives but does not reconnect through quickly:

1. Contributor A is assigned issue X; last progress report at t=0.
2. A's machine sleeps, or its network/VPN drops, at t≈2 (the relay process stays alive; its CLI keeps the task locally).
3. At t=10 the release cooldown lapses.
4. At t=13 contributor B sends ready → selectTask offers X to B (not in cooldown, no live holder).
5. At t=15 A's relay reconnects, re-asserts X, lookupLease matches (lease alive until t=30), A resumes with a fresh token.
6. A and B both work X and both open a PR.

Reproduction

TestAudit_ResumeAfterReleaseCooldownLapsesDoubleAssigns (run against the real handlers via go test -overlay) drives exactly this ordering and asserts both connections end up holding X. It passes = the double-assignment reproduces.

Why it happens

The disconnect hedge (#2356) and the resumable lease (#4260/#5681) were designed against different timescales — one to bridge the reconnect backoff, the other to let a long-running task survive a drop — and their windows were never reconciled. A lease that is still re-adoptable is, in effect, a hold; but only the live-connection scan and the cooldown are consulted when deciding whether to re-offer an issue, so a leased-but-not-live issue falls through.

What "fixed" looks like

An issue that is still re-adoptable under a live lease is not offered to another contributor. Either the release cooldown lasts at least as long as the lease remains re-adoptable, or selectTask treats an unexpired lease held by a not-currently-connected identity as an in-flight exclusion (the way the post-restart grace already treats a restored lease), so the offerable window and the resumable window cannot overlap.

Impact

Duplicate PRs and wasted contributor work — the exact failure #2356 exists to prevent — for any outage between the release-cooldown length and the lease TTL. The hub log shows both a "task assigned" (to B) and a "task resumed" (by A) for the same issue key.

Suggested regression coverage

A hub test: assign an issue, disconnect, advance the clock past the release cooldown but within the lease TTL, have a second contributor ready, and assert the issue is not re-offered while the lease is live; then reconnect the first relay and assert it still resumes.

Affected components

• Repo hivecommons/hive, package src/pkg/dashboard.
• contribute_ws.go: releaseOnDisconnect (~1780), the bookReleaseCooldown call (~1870).
• contribute_cooldowns.go: failedTaskCooldownMinutes (19), the failure-cooldown check.
• contribute_leases.go: leaseTTL (64), lookupLease.
• contribute_select.go: the activeIssues live-connection scan (~337) that is the only in-flight exclusion.

[github-actions]

@Danathar this issue was closed by @clubanderson as completed.

If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.