feat: Add host-driven WASI event loop experiment

[krodak]

Overview

This draft tests the event-loop model suggested by akmorrison on upstream PR swiftlang#953. The existing WASI backend drains work from a queue poke. In the registered mode introduced here, a poke requests one later host callback, dispatch_async returns, and the host calls back into Dispatch to perform work.

The experiment is opt-in and targets the combined WASI branch so we can compare both models without rewriting the existing suite. Unregistered builds keep the current eager behavior. The SPI names and callback shape are open for discussion.

Review focus

• Is callback registration the right boundary between libdispatch and an embedded runtime?
• Should the host tell perform whether a scheduler callback or timer caused the turn, or should the SPI hide that distinction?
• Is a drain-phase budget useful for this proof, given that a manager or main-queue phase can drain a captured snapshot?
• If this direction holds, registered host-loop mode would replace eager draining and remove the poke-defer hooks from the upstream design.

Links

• Upstream feedback: feat: prepare shared code for cooperative single-threaded event backends swiftlang/swift-corelibs-libdispatch#953 (comment)
• Combined WASI port: feat: Full combined set of changes to add Wasm support to libDispatch #3
• Design note: https://github.com/PassiveLogic/swift-corelibs-libdispatch/blob/5db7d4e08596ed084087965cd201fef78ab83fd3/docs/wasi-host-event-loop/README.md

What changed

• private/private.h - adds an experimental WASI-only SPI for scheduler registration, step-budgeted perform calls, and relative timer delays.
• src/event/event_wasi.c - coalesces host wakeups, routes registered pokes out to the host, preserves eager draining when unregistered, and distinguishes scheduler turns from timer turns so one cannot consume the other's latch.
• tests/wasm/host-event-loop.c and tests/wasm/run-wasi-host-event-loop-test.mjs - add a Node-driven WASI reactor that proves later-turn execution, initial wakeup coalescing, timer-only progress, retained scheduler turns, and named failures for inline callbacks and late registration.
• docs/wasi-host-event-loop/ - documents the experiment and includes the Mermaid sources and rendered PNG diagrams.

Fd readiness remains outside this experiment. Perform can harvest readiness already visible to WASI, but an idle JavaScript host still needs runtime-specific notification when an fd becomes ready. Blocking waits also retain the combined branch's current cooperative behavior.

Tested

• cmake -S . -B build-wasi -G Ninja -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/WASI.cmake -DSWIFT_WASI_TOOLCHAIN_PATH="$HOME/Library/Developer/Toolchains/swift-6.3.3-RELEASE.xctoolchain" -DSWIFT_WASI_SDK_PATH="$HOME/Library/org.swift.swiftpm/swift-sdks/swift-6.3.3-RELEASE_wasm.artifactbundle/swift-6.3.3-RELEASE_wasm/wasm32-unknown-wasip1" -DENABLE_SWIFT=YES -DBUILD_TESTING=ON
• cmake --build build-wasi
• ctest --test-dir build-wasi --output-on-failure
• ctest --test-dir build-wasi -R '^dispatch_wasi_host_event_loop$' --repeat until-fail:200 --output-on-failure

[scottmarchant]

Hi @krodak, thanks for this. It maps well onto what akmorrison asked for, and it is the direction we are taking.

I pushed two commits on top of yours (457a699 and c32b54a). They make your model the only submission model instead of an opt-in. Revert either one if you disagree with any part of it.

What changed:

• A poke only records pending work now. It never drains on the submitting stack. Pending work runs at a pump point: a top-level blocking wait, dispatch_main(), or a host turn through your _dispatch_wasi_event_loop_perform().
• The eight poke-defer hook sites, the sync funnel split, and the defer depth counter are gone. A poke never runs client code under a caller-held lock, so the hooks have no purpose. The shared files return to their upstream shape at those sites.
• Registration no longer has to happen before the first Dispatch use. Work recorded before registration is handed to the scheduler at registration. A pump that stops with runnable work left, for example a satisfied or timed-out wait, hands that work to the host too.
• Your SPI shape is unchanged: the drain-phase budget and the consumes_scheduled_turn flag stay as you wrote them.
• Tests: eager-drain.c became deferred-submission.c. Eleven focused tests gained a pump where they relied on eager progress. The late-registration reactor test is now a positive test, and a new reactor test covers registration from inside a running work item. The suite is 57 of 57 with the Swift 6.3.1 toolchain and SDK. The host loop tests pass 50 repeats.
• Docs: the README semantics section, the wait-pumping audit, and your design note now describe this as the model. The stale execution-model PNG became an inline Mermaid block.

One design rule is worth stating: on one thread every queue behaves like the Darwin main queue. Work runs when control returns to the host or during a Dispatch blocking wait. Code that submits work and then spins, sleeps, or blocks in a plain read() sees no progress, exactly as it would on the Darwin main thread.

My plan from here:

1. If you agree, please merge this PR into feat: Full combined set of changes to add Wasm support to libDispatch #3. I will rerun the suite there.
2. I re-scope feat: prepare shared code for cooperative single-threaded event backends swiftlang/swift-corelibs-libdispatch#953 in place: drop the hooks, keep the main-queue drain hoist. That is 14 lines in one file.
3. I reply to akmorrison: agree, map the model onto the CFRunLoop _4CF seam, and ask whether a registered scheduler callback plus a next-deadline query is an acceptable WASI arm of dispatch_runloop_handle_t.
4. Platform glue follows: the JavaScriptKit runtime hookup and the Khasm bootstrap registration, so consumer code stays the same on JS hosts.
5. Fork PR Prepare shared code for cooperative single-threaded event backends #5 gets the same re-scope or a pointer and a close.

Not done yet: a Linux neutrality build on this branch. The shared-code change is a return to upstream text, so the risk is low, but I will run it before swiftlang#953 moves.

[scottmarchant]

It's probably fine to merge these pngs and diagrams to #3, but I'll scrub them out when I update the upstream PR.

I pushed some updated, there is a comment by AI about the changes and my plan. Please feel free to look through and revise

[krodak]

@scottmarchant changes looks good, merged to #3 , let me know if you needed any help with propagating these to upstream PR and 🤞🏻 for further review process 😄