fix(transport): eliminate startup hangs and endpoint hijacking in host-emulation IPC

[nehalkpatel]

Context

This started as a question about ZMQ slow-joiner exposure. The classic slow-joiner bug is not present — there is no PUB/SUB anywhere in this project. Every socket on both sides is ZMQ_PAIR, which has no subscriptions to propagate and blocks in its mute state rather than discarding, so a send with no peer surfaces as kTimeout instead of vanishing while Send() reports success. No XPUB/handshake work was warranted.

What the investigation did surface was a family of related defects: startup paths that hang, state that lies, and two processes silently sharing one endpoint.

What was wrong

Create() could hang forever. bind_cv_.wait() had no deadline, so when the server thread failed to bind — a bad path, a second instance — the catch-all swallowed the throw, the thread exited, and the constructor waited on a predicate that could never become true.

The constructor could std::terminate. If connect() threw, the exception escaped while server_thread_ was joinable, and ~std::thread aborts the process before Create()'s handler runs.

kConnected was a guess. It was set immediately after the asynchronous connect(), so it only ever meant "the constructor ran". WaitForConnection looped while kConnecting and therefore never iterated, leaving connect_timeout dead. shutdown_timeout was never read at all.

Endpoints could be silently stolen. This one overturned the original plan's premise. We assumed bind() fails with EADDRINUSE when an ipc path exists. It doesn't — libzmq unlinks the path first, unconditionally. Measured on this tree: bind over a stale file succeeds unaided, and bind over a path a live process is listening on also succeeds. libzmq removes the live owner's rendezvous name and takes its place. The original keeps its existing connections, because the inode outlives the name, but every later connect() reaches the thief. A second app instance, or a unit test run while the emulator is up, splits the bus in two with no error anywhere.

So stale-file cleanup was solving a non-problem while the real defect ran the other way. The emulator was worse still: it unlinked the socket file itself, unconditionally, making displacement certain rather than merely possible.

What changed

The state machine now says what it actually knows. kReady means "our socket is bound and connect() was issued", and documents that it claims nothing about the peer — peer liveness stays where it is genuinely observable, in the result of the Send you attempted. connect_timeout became startup_timeout with a real consumer; shutdown_timeout was deleted, since std::thread has no timed join and nothing could honour it.

The constructor is now total: never throws, never blocks past startup_timeout, always yields a queryable object. That is what makes the public constructor safe to leave public — a direct caller who ignores StartupStatus() gets a kFailed object whose every call returns kInvalidState, which is loud and bounded rather than an infinite hang.

Endpoint ownership is guarded by an flock-based EndpointLock, with a connect(2) liveness probe behind it for an owner that holds no lock. flock is the right primitive for two reasons: it is kernel-arbitrated, so there is no window to lose, and the lock lives on the open file description so the kernel drops it on process death — a SIGKILLed run leaves nothing stale. An O_EXCL marker would have needed its own liveness check, recreating the problem one level up. The lock file is deliberately never unlinked; removing it would let two processes hold locks on different inodes and both proceed.

The same guards are implemented on the Python side, with matching lock-file naming — that is load-bearing, not cosmetic. If the two disagreed on the path they would each lock a different file and exclude nothing.

The race, measured

A probe followed by a bind is two syscalls with a window between them. Contending it properly needs fork plus a shared-memory spin barrier — launching processes from a shell spreads arrivals over milliseconds and the first binder always wins, which makes the race look absent:

	winners per run (12 contenders)
probe only	1, 3, 4, 4, 5, 5, 6, 7
probe + flock	1, 1, 1, 1, 1

Every winner above 1 is a process that believes it owns an endpoint libzmq has already taken from it. ConcurrentBindsProduceExactlyOneOwner makes this permanent.

Tests

Fixtures now synchronise on conditions instead of sleeps. Binds happen on the test thread before the serving thread exists (ordering, not timing); the post-construct sleep became a probe Send, which on PAIR succeeds only once a pipe exists and so is the readiness signal. _wait_for_process_ready had no success exit at all — it slept out its full timeout every call and treated "did not die in 1.1s" as ready; it now waits for the app's ipc file, which the transport binds inside Create().

Two genuine bugs fell out along the way: handler_called/received_data were a data race across the server thread regardless of any sleep, and test_zmq_transport's teardown sleep was load-bearing — it terminated the context before joining, and a context terminated under zmq::poll throws ETERM out of a thread with no handler.

Three C++ tests were verified against the pre-change tree by reverting the production files:

Test	Before
CreateFailsFastWhenBindEndpointIsUnbindable	hangs (watchdog fires at 10s)
CreateRefusesToStealEndpointFromLiveOwner	fails — has_value() true, hijack succeeded
CreateSucceedsOverStaleSocketFile	passes — not a regression test, and says so

The third is kept as a guard against the new probe over-rejecting: reading "file exists" as "owned" would refuse to start after any crash. On the Python side, test_second_emulator_refuses_to_steal_endpoint was likewise verified to fail against the previous unlink behaviour.

Verification

• 33 tests green (was 29) in both host-debug and host-release
• 30 consecutive serial unit runs clean
• ctest -j8 clean across repeated runs — it was already red before this work, with the same fixtures failing from silent mutual corruption; per-pid endpoints fix that
• ruff and mypy --strict clean across 13 files
• Total test time 10.2s -> 4.1s, from removing the sleeps

Not included

Send's retry loop can never retry — total_timeout and SNDTIMEO are both 1000ms, so the first EAGAIN arrives at or after the deadline and max_attempts{3} is dead config. Filed as #30 and deliberately left out as a send-path defect rather than a startup one.

Also still open: the probe->bind sequence remains non-atomic for an owner that holds no lock, and kReady is never invalidated if the emulator dies mid-run.

🤖 Generated with Claude Code