Skip to content

perf(common): notify RingChannel consumers via a lock-free park stack - #1649

Open
Coldwings wants to merge 1 commit into
alibaba:mainfrom
Coldwings:ringchannel_park_slot
Open

Coldwings wants to merge 1 commit into
alibaba:mainfrom
Coldwings:ringchannel_park_slot

Conversation

@Coldwings

Copy link
Copy Markdown
Collaborator

Problem

RingChannel notifies its idle consumers through a photon::semaphore. That path
does not scale with the number of vCPUs:

  • semaphore::signal() holds one
    per-channel spinlock across try_resume(), which calls prelocked_thread_interrupt().
    For a consumer parked on another vCPU that is a cancel_wait() eventfd write syscall
    executed inside the critical section
    .
  • A consumer returning from wait_defer() re-acquires that very same lock.
  • Every recv() that misses did two seq_cst RMWs on the shared idler counter.

So all notifications of a channel serialize behind a single lock whose holder can be
preempted while everybody else spins on it. WorkPool dispatch throughput therefore falls
as vCPUs are added.

Change

A park slot per sleeping consumer, linked into a lock-free idle stack (ParkStack):

  • A producer's fast path is a fence plus a relaxed load of the stack top: no store to
    shared state, no lock, nothing to contend on while consumers are busy.
  • A wake-up is one exchange() of the whole stack plus a thread_interrupt() issued
    outside of any lock, so N producers wake N consumers in parallel.
  • A consumer touches the shared word only when it actually parks, not on every recv().

FlexRingChannel gets the same treatment. The 100ms self-wake is kept as a safety net
only: it re-checks the queue and re-arms the slot, and correctness does not depend on it.

Why no notification is lost

  1. Dekker. The producer fences between its push and its idle() load; the consumer
    publishes its slot with a seq_cst RMW and re-checks the queue right afterwards. The
    two cannot miss each other ([atomics.order]/4 needs the fence on both sides, hence the
    one after publish()).
  2. Publish-in-defer. prepare_usleep() does not inspect error_number, so an
    interrupt arriving before the sleep commits would be dropped. The slot is promoted to
    COMMITTED from the defer callback of thread_usleep_defer(), and only a COMMITTED
    slot may be interrupted. A claimer that finds it not yet committed leaves the wake-up to
    the owner's own defer callback.
  3. Slot lifetime. A slot is a stack frame, so its owner never returns from park()
    while a claimer may still touch it. The claimer's last write is its CLAIMED store; the
    owner waits for it, escalating pause → thread_yieldsched_yield, because if the
    store has not landed the claimer is not on a CPU and spinning only keeps it away. That
    escalation alone cuts hand-off stalls of ~1.5ms (0.3% of parks, but 15-33% of the wall
    time of the sync WorkPool cases) down to microseconds.

Numbers

Interleaved A/B of two binaries on the same host (each round runs both, so machine drift
cannot favour either side). perf_workpool --fires=40000, sync/StdContext case, median of 4:

vCPUs 4 8 16 32
QPS before 40413 18416 9162 6898
QPS after 66126 111856 141728 134764
ns/dispatch before 9042 15860 36472 49332
ns/dispatch after 5638 2478 1572 1512

The sync/PhotonContext case has the same shape (38214 → 53454 QPS at 4 vCPUs,
23268 → 136678 at 16). Notice the direction: before, the curve falls from 8 vCPUs on;
after, it rises until the host runs out of cores (11 here) and then flattens.

perf-ringchannel, median of 4, N producers × N consumers:

N=4 N=16 N=32
burst fan-out, ns/item 1719 → 1426 2998 → 682 4594 → 329
multi-producer contention, ns/item 308 → 176 1632 → 287 1981 → 586
cross-vCPU wake-up, ns 12293 → 1734 12367 → 1266 9816 → 1650

The whole perf_workpool run also spends 39% less user CPU (2.96s → 1.85s) and 24%
less wall time.

Honest about the costs:

  • Fire-and-forget dispatch on few vCPUs regresses ~30% (4 vCPUs, paced producer). Root
    cause: the semaphore version hardly ever really slept there — leftover m_count tokens
    made wait() return immediately — so it bought throughput by burning CPU. This turns
    into a 1.5x-3.4x gain by 16 vCPUs.
  • Same-vCPU send+recv on one channel is ~20% slower per round trip (publish CAS plus
    the CLAIMING/CLAIMED hand-off). That shape means a thread notifying itself, which no real
    user does.

Tests

  • New test-ringchannel-notify: six cases (strict ping-pong, ping-pong racing the safety
    net, MPMC burst, single producer against 8 consumers, external thread_interrupt during
    a park, same-vCPU pairing). They run the channel with the safety net disabled, so a
    lost notification hangs the case and an OS-thread watchdog reports it, instead of hiding
    as a 100ms hiccup. A producer paces itself against an empty queue so that every item has
    to pay the full notification path.
  • Verified the test has teeth by mutation: removing the re-check after publish() is
    caught within ~30k rounds; committing the slot before the sleep instead of from the defer
    callback within ~800.
  • New perf-ringchannel (not registered with ctest) for the numbers above.
  • test-lockfree, test-executor-* (including test-executor-burst-drain, which asserts
    no notification accumulates over a burst), test-go-channel, test-workpool-fanout all
    pass.

Backport

Recommended for at least release/0.9 (labelled need-backport so the cascade can
carry it further if maintainers want 0.8 too). Rationale: this is a scalability fix for
anything built on WorkPool / Executor with 8+ vCPUs, and it changes no interface that
affects source compatibility — send(), recv() and notification_pending() keep their
signatures, and notification_pending() keeps its meaning of "wake-ups issued but not yet
observed". The class is header-only, so its layout does change and consumers must be
rebuilt against matching headers, which any photon upgrade already requires.

RingChannel woke its idle consumers through a photon::semaphore, whose
signal() holds one per-channel spinlock across try_resume(), which in
turn calls prelocked_thread_interrupt() -- so a cross-vCPU wake-up costs
an eventfd write syscall inside the critical section. Every consumer
returning from wait_defer() re-acquires that same lock, and every recv()
miss did two seq_cst RMWs on the shared `idler` counter. All
notifications of a channel thus serialize behind one lock whose holder
may be preempted while everybody else spins on it, so dispatch
throughput fell as vCPUs were added instead of rising.

Replace it with a park slot per sleeping consumer, linked into a
lock-free idle stack. A producer's fast path is a fence and a relaxed
load of the stack top: no store to shared state, no lock. A wake-up is
one exchange() of the whole stack plus a thread_interrupt() issued
outside of any lock, so N producers can wake N consumers in parallel.

Not losing a notification rests on three invariants:

- Dekker: the producer fences between its push and its idle() load,
  while the consumer publishes its slot with a seq_cst RMW and re-checks
  the queue right afterwards. The two can not miss each other.
- Publish-in-defer: prepare_usleep() does not inspect error_number, so
  an interrupt arriving before the sleep commits would be dropped. The
  slot is therefore promoted to COMMITTED from the defer callback of
  thread_usleep_defer(), and only a COMMITTED slot may be interrupted.
- Slot lifetime: a slot is a stack frame, so its owner never returns
  from park() while a claimer might still touch it. The claimer's last
  write is its CLAIMED store, and the owner waits for it, escalating
  from pause to thread_yield to sched_yield: if that store has not
  landed yet, the claimer is not on a CPU, and spinning would only keep
  it away. The escalation alone cuts hand-off stalls of ~1.5ms (0.3% of
  all parks, yet 15-33% of the wall time of the synchronous WorkPool
  cases) down to microseconds.

The 100ms self-wake of a parked consumer is kept as a safety net only:
it re-checks the queue and re-arms the slot. Correctness does not
depend on it, which is what lets the new test disable it.

Interleaved A/B on one host, perf_workpool --fires=40000, the
sync/StdContext case, median of 4 runs:

  vCPUs         4       8      16      32
  before    40413   18416    9162    6898  QPS
  after     66126  111856  141728  134764  QPS
  before     9042   15860   36472   49332  ns/dispatch
  after      5638    2478    1572    1512  ns/dispatch

Cross-vCPU wake-up latency improves 4.6x, burst fan-out 1.2x with 4
consumers and 14x with 32, and the whole benchmark spends 39% less user
CPU. Fire-and-forget dispatch on few vCPUs is the one case that
regresses (~30% at 4 vCPUs), because the semaphore version hardly ever
really slept there; it turns into a 1.5x to 3.4x gain by 16 vCPUs.

Add test-ringchannel-notify, which drives the channel with its safety
net disabled, so that a lost notification hangs the case and is
reported, instead of hiding as a 100ms hiccup. Its teeth were verified
by mutation: dropping the re-check after publish, or committing the
slot before the sleep instead of from the defer callback, are both
caught within a few thousand rounds. Add perf-ringchannel to measure
the notification paths in isolation.

No public interface changes: send(), recv() and notification_pending()
keep their signatures, and notification_pending() keeps its meaning of
wake-ups issued but not yet observed. Being header-only, the class does
change layout, so consumers must be rebuilt against matching headers,
as any photon upgrade already requires.
@Coldwings Coldwings added the need-backport A PR that should be back-ported to prior release branches (release/*) label Sep 15, 2026
@Coldwings
Coldwings requested review from EricHuangqx, beef9999, lihuiba and liulanzheng and a lite review from Copilot and removed request for Copilot September 15, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

need-backport A PR that should be back-ported to prior release branches (release/*)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant