Conversation
…sfy any waiter
signal() used to take splock and walk the wait queue on every single call, even
when the count in hand could satisfy nobody. With several vCPUs signalling one
semaphore -- the shape a WorkPool produces -- that spinlock is where the time
goes, and the walk drags in the waiters' thread structs, which are hot lines of
other vCPUs.
semaphore now keeps m_min_wait, a lower bound of what any waiter needs, or -1
when nobody waits. signal() adds its count and returns immediately as long as
the total stays below that bound. wait_interruptible() gets a lock-free fast
path as well, for the case where the count is already there.
The two sides form a Dekker pair, and both are sequentially consistent, so at
least one of them observes the other:
signal(): m_count.fetch_add(n) then load(m_min_wait)
wait(): m_min_wait.store(count) then load(m_count)
The invariant is that m_min_wait never exceeds the requirement of any sleeping
waiter. Being too low only costs a needless walk, being too high would lose a
wake-up, so it is only raised back to -1 while holding splock with the queue
observed empty, and every waiter re-publishes on each iteration of its loop --
it may have been resumed and dequeued, but not yet run, while another waiter
drained the queue and reset the bound. The publish-then-enqueue span is covered
by splock, since wait_interruptible() holds it until prepare_usleep() has
pushed the waiter onto the queue.
The wait-side fast path lets a newcomer take a count that queued waiters are
waiting for. That does not change the semantics: the queue was never strict
FIFO -- a resumed waiter has to win try_subtract() against everybody else
anyway, and the out-of-order mode reorders by design.
Measured with thread/test/perf_semaphore.cpp, 8 signalling vCPUs x 50k ops,
4 waiters, 500ns of work between signals, aarch64, MinSizeRel, median of 3:
wall signal() avg signal() max
no waiter 53.5 -> 42.6ms 348 -> 171ns
batch 98.1 -> 36.1ms 1807 -> 179ns 127us -> 0.7us
stream 261.0 -> 61.6ms 5777 -> 636ns 473us -> 142us
With zero work between signals the aggregate throughput goes the other way,
2ns/op to 15ns/op: serialising on the spinlock lets one core keep the counter in
its own L1 and run long uncontended bursts, while the fast path has all eight
adding to the same line fairly. That burst throughput comes with a 1.4ms
worst-case signal() and a 10x spread across vCPUs, and it evaporates as soon as
there is any work between signals, which is the case this optimisation targets.
Once m_min_wait keeps the hopeless signals out, the ones that remain all pile
onto splock and then onto the same wait queue, each dragging in the waiters'
thread structs. The work is serialized anyway, and it is redundant: whoever is
in the resume path re-reads m_count, so he covers the counts of everybody else.
signal() now claims the path with a RESUMING bit and does the walk for all; a
signaler that finds it taken sets RECHECK instead and leaves. The holder does
not leave until he has read m_count with RECHECK observed clear, and since one
may still set it between that last look and the release, the release itself
reports the bit back and the holder takes the role again.
Every one of these transitions is a read-modify-write on m_resume_state, so
they are totally ordered with each other, and so is the fetch_add on m_count.
RECHECK is cleared before m_count is read, never after. A signaler that leaves
the work to somebody else therefore always has a resume behind it that observes
its count: either the current holder loops again, or -- if he has already gone
past his last look -- his release reports RECHECK and he comes back, or a third
signaler has claimed the path since, and that claim is ordered after our
RECHECK, hence after our count.
The trade is that the holder does the work of all comers, so a signaler can
stay in the loop as long as others keep feeding it. That is the point: it is
the same total work, done on one core without the convoy.
thread/test/perf_semaphore.cpp, the stream case (8 signaling vCPUs x 50k ops,
4 waiters, 500ns of work between signals), interleaved A/B, 10 rounds each,
aarch64, MinSizeRel:
wall (median) signal() avg signal() worst
before 58.3ms 597ns 230us
after 48.2ms 348ns 17us
The two wall-time ranges barely overlap (53.3-72.3ms against 43.6-57.1ms), and
the worst case of signal() went from 43-230us in every round down to 3-17us.
One 1.35ms sample did show up in an earlier three-round run, which is what a
long combining stretch looks like; ten rounds did not reproduce it.
test-throttle fails on this machine regardless of these changes -- its loss
ladder is dominated by thread_usleep() granularity, and a three-way run over
the ooo fix alone, m_min_wait, and this commit yields the same numbers within
noise (0.136/0.145/0.3532/0.6587/0.932 against 0.086/0.134/0.3728/0.6689/0.931).
Coldwings
marked this pull request as draft
September 16, 2026 09:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent optimizations to
photon::semaphorethat cut the signal-sidespinlock convoy seen when several vCPUs signal one semaphore -- the shape a
WorkPool produces. Based directly on
main; does not depend on #1650.Changes
Skip the resume path when the count cannot satisfy any waiter (
m_min_wait):signal()keeps a lower bound of what any waiter needs (or -1 when nobodywaits), adds its count and returns immediately while the total stays below that
bound, without taking
splockor walking the queue.wait_interruptible()gets a matching lock-free fast path. The two sides form a Dekker pair, both
sequentially consistent, so at least one observes the other and no wake-up is
lost:
Let one signaler at a time walk the resume path (combining): once
m_min_waitkeeps the hopeless signals out, the ones that remain all pile ontosplockand the same queue, redundantly -- whoever is in the resume pathre-reads
m_countand covers the others' counts. A signaler now claims thepath with a
RESUMINGbit and serves everyone; one that finds it taken setsRECHECKand leaves. The holder does not release until it has readm_countwith
RECHECKobserved clear, and the release itself reports the bit back incase someone set it at the last moment. Every transition is a read-modify-write
on
m_resume_state, totally ordered with thefetch_addonm_count.Adds
thread/test/perf_semaphore.cpp, a benchmark (NO_REGISTER).Motivation
With several vCPUs signalling one semaphore,
signal()used to takesplockandwalk the wait queue on every single call, even when the count in hand could
satisfy nobody, dragging in the waiters' thread structs -- hot lines of other
vCPUs. That spinlock is where the time goes.
Measured with
thread/test/perf_semaphore.cpp, 8 signalling vCPUs x 50k ops,4 waiters, 500ns of work between signals, aarch64, MinSizeRel, median of 3:
With zero work between signals the aggregate throughput goes the other way:
serialising on the spinlock lets one core keep the counter in its own L1 and run
long uncontended bursts, while the fast path has all eight adding to the same
line fairly. That burst throughput comes with a much worse worst-case
signal()and a large spread across vCPUs, and it evaporates as soon as there is any work
between signals, which is the case this optimisation targets.
References
on
mainand touches regions ofsemaphore::try_resume()disjoint from thatfix, so the two can land in either order.