Skip to content

fix(thread): stop semaphore's out-of-order resume from deadlocking on the wait queue lock - #1650

Open
Coldwings wants to merge 1 commit into
alibaba:mainfrom
Coldwings:fix-semaphore-ooo-resume-deadlock
Open

Coldwings wants to merge 1 commit into
alibaba:mainfrom
Coldwings:fix-semaphore-ooo-resume-deadlock

Conversation

@Coldwings

Copy link
Copy Markdown
Collaborator

Summary

photon::semaphore deadlocks against itself in the out-of-order resume path, taking down every vCPU that touches the semaphore afterwards. The path is only reachable for a semaphore constructed as semaphore(count, /*in_order_resume=*/false), which nothing in the tree currently does, so no in-tree user is affected today — but the mode is public API and the failure is deterministic, not probabilistic.

Changes

  • semaphore::try_resume(): the out-of-order walk now tells the resume path that the wait queue lock is already held, and remembers the successor of the node it is about to resume.
  • thread::dequeue_ready_atomic() and prelocked_thread_interrupt(): new waitq_locked parameter, defaulting to false, using the SCOPED_LOCK(x, cond * 2) idiom already used by prepare_usleep(). The other three call sites keep their behaviour byte for byte.
  • New thread/test/test-semaphore-lost-wakeup.cpp (9 cases), registered in thread/test/CMakeLists.txt.

Motivation

Two defects sit in the same loop:

  1. Self-deadlock. When the queue head demands more than the count in hand, try_resume() walks the queue under q.lock looking for a smaller waiter behind it. Resuming that waiter goes through prelocked_thread_interrupt()thread::dequeue_ready_atomic(), which locks the very same q.lock to erase the thread from the queue. photon::spinlock is not recursive, so the walk spins on a lock it already owns — while still holding the semaphore's splock, so every other vCPU entering signal() or wait() piles up behind it.

  2. The walk could never advance. It stepped with th = th->next() after the resume, but a resumed thread has already been erased from the queue, and erase()remove_from_list() leaves the node's links pointing at itself.

Why this has gone unnoticed: the branch is guarded by m_ooo_resume, and the default is in-order. The in-order branch resumes through ScopedLockHead, which holds only th->lock, so it is safe. Even photon::throttle, whose waiters ask for wildly different counts (bytes) and therefore do leave the head out of reach, uses the default constructor and never enters the out-of-order branch.

Verification

Built and tested on aarch64 Linux, MinSizeRel:

unpatched main with this fix
SemaphoreLostWakeup.ooo_bypass hangs (SIGKILL by timeout) pass
SemaphoreLostWakeup.mixed_counts_ooo hangs (SIGKILL by timeout) pass
whole new suite 9/9 pass
test-thread 69/69 pass
test-multi-vcpu-locking, test-std-compat, test-go-channel, test-workpool-fanout, test-sleepq-stale-idx, test-pool pass

ooo_bypass is deterministic and single-vCPU — it queues a waiter for 10, then one for 1, and signals 1 — so it also proves the out-of-order branch is actually entered, rather than relying on a race being hit by chance.

A regression here is a hang, not a wrong value, so the suite carries a watchdog on a plain OS thread rather than an in-coroutine deadline: under a spinlock convoy the photon scheduler may never run the main photon thread again, which was observed while diagnosing this bug and makes any coroutine-side deadline useless.

References

  • semaphore::try_resume() in thread/thread.cpp
  • semaphore(uint64_t count, bool in_order_resume) in thread/thread.h
  • photon::spinlock (non-recursive TTAS) in thread/thread.h

… the wait queue lock

semaphore::try_resume() walks the wait queue under q.lock when the head demands
more than the count in hand, looking for a smaller waiter behind it. Resuming
that waiter goes through prelocked_thread_interrupt() ->
thread::dequeue_ready_atomic(), which locks the very same q.lock to erase the
thread from the queue. photon::spinlock is not recursive, so the walk deadlocks
against itself -- while still holding the semaphore's splock, which takes down
every vCPU that touches the semaphore afterwards.

The same loop also advanced with th = th->next() after the resume, but a resumed
thread has already been erased from the queue, and erase() leaves its links
pointing at itself, so the walk could never have moved on.

dequeue_ready_atomic() and prelocked_thread_interrupt() now take a waitq_locked
flag, following the SCOPED_LOCK(x, cond * 2) idiom already used by
prepare_usleep(), and the out-of-order walk remembers the successor before
resuming. The flag defaults to false, so the other three call sites keep their
behaviour.

The path is only reachable with a semaphore constructed as
semaphore(count, /*in_order_resume=*/false), which nothing in the tree does --
the default is in-order, and the in-order branch resumes through
ScopedLockHead, holding only th->lock. That is why this has gone unnoticed:
even throttle, whose waiters ask for wildly different counts, never enters it.

The new test hangs deterministically without this fix, on a single vCPU:
ooo_bypass queues a waiter for 10, then one for 1, and signals 1. The remaining
cases hunt for waiters that never wake up under multi-vCPU traffic, in-order and
out-of-order alike. A regression here is a hang rather than a wrong value, so
the suite carries a plain OS-thread watchdog: under a spinlock convoy the photon
scheduler may never run the main thread again, which makes any in-coroutine
deadline useless.
@Coldwings Coldwings added bugfix A PR that fixes a bug need-backport A PR that should be back-ported to prior release branches (release/*) labels Sep 15, 2026
@Coldwings
Coldwings requested a review from lihuiba September 15, 2026 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix A PR that fixes a bug need-backport A PR that should be back-ported to prior release branches (release/*)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant