Skip to content

fix(transport): make ZmqTransport::Send actually retry - #33

Merged
nehalkpatel merged 1 commit into
mainfrom
fix/send-retry-loop
Aug 27, 2026
Merged

fix(transport): make ZmqTransport::Send actually retry#33
nehalkpatel merged 1 commit into
mainfrom
fix/send-retry-loop

Conversation

@nehalkpatel

Copy link
Copy Markdown
Owner

Fixes #30.

Two causes, not one

The issue's diagnosis was right that the retry path was unreachable, but there were two independent causes — and fixing only the timeouts would not have helped.

1. The timeout was never classified as retryable. cppzmq's send() reports an expired ZMQ_SNDTIMEO by returning an empty result, and throws error_t only for everything else (zmq.hpp:1967-1976). Send() looked for EAGAIN exclusively inside a catch block and treated the falsy return as a hard failure, so a send timeout came back as kOperationFailed and never reached the loop's deadline check at all.

2. The timeouts left no room, as reported. send_timeout bounds one attempt, retry.total_timeout bounds all of them, and both defaulted to 1000ms.

Changes

  • TrySendOnce classifies one attempt as kSent/kWouldBlock/kFailed, catching the timeout in both forms ZMQ reports it. Send() is rewritten around that, and every exhausted path now returns kTimeout — a blocked PAIR socket means the peer never took the message either way.
  • ClampSendTimeoutToRetryBudget(), called from the constructor, enforces max_attempts * send_timeout + (max_attempts-1) * retry_delay <= total_timeout by shrinking send_timeout and warning. It also floors max_attempts at 1, which otherwise skipped the loop entirely.
  • Default send_timeout 1000ms → 300ms, so the defaults satisfy the invariant unclamped: 3×300 + 2×10 = 920ms ≤ 1000ms.

Clamping rather than rejecting the config: a caller asking for three attempts within a second has said something coherent, and the arithmetic that makes it fit is ours to do rather than grounds for failing startup.

A correction to the issue, and to the header

A peerless PAIR socket does not go mute, contrary to the issue's closing note and the header's own comments. connect() creates the outbound pipe whether or not the far end is reachable, and libzmq queues into it, so sends to nobody succeed until ZMQ_SNDHWM (1000) messages are outstanding. Measured directly. The stale comments are corrected here.

That is also why the tests below fill the queue rather than simply pointing at an unbound endpoint — the obvious setup would have passed against the broken code.

Tests

Two tests, each asserting the error, the retry count, and the elapsed time, so no single wrong answer satisfies all three:

  • SendAttemptsEveryRetryWhenSocketBlocks — measured 326ms against a predicted 3×100 + 2×10 = 320ms.
  • SendRetriesWhenSendTimeoutClaimsTheWholeBudget — the shape of the config that shipped. Measured 605ms against a predicted 3×193 + 2×10 = 599ms, bounded both ways so it distinguishes shortened attempts from full-length ones.

Against the unfixed code both fail, with kOperationFailed and zero retries.

Verification

cmake --workflow --preset=host-debug and --preset=host-release both green — 35/35 tests, including the Python integration suite. tools/format.sh --check clean; clang-tidy clean via the build.

🤖 Generated with Claude Code

The retry loop was unreachable, for two independent reasons.

cppzmq's send() reports an expired ZMQ_SNDTIMEO by returning an EMPTY
result and throws error_t only for everything else. Send() looked for
EAGAIN exclusively inside a catch block and treated the falsy return as
a hard failure, so a send timeout came back as kOperationFailed without
ever reaching the loop's deadline check. Classify the attempt instead,
in a TrySendOnce helper that catches the timeout in both the forms ZMQ
reports it.

Independently, the timeouts left no room to retry: send_timeout bounds
ONE attempt and retry.total_timeout bounds all of them, and both
defaulted to 1000ms, so the first EAGAIN arrived at the deadline and
max_attempts and retry_delay described behaviour that could not happen.
Size the default per-attempt slice to fit (3 * 300ms + 2 * 10ms <=
1000ms), and enforce the invariant at construction by shrinking
send_timeout for any caller-supplied config that violates it. Clamping
beats rejecting: asking for three attempts within a second is coherent,
and the arithmetic that makes it fit is ours to do rather than grounds
for failing startup.

Two tests, both driving real backpressure by filling the send queue past
ZMQ_SNDHWM, since -- contrary to what the header used to claim -- a
peerless PAIR socket does not go mute. connect() creates the outbound
pipe whether or not the far end is reachable and libzmq queues into it,
so sends to nobody succeed until the high-water mark. Corrected those
comments too. Each test asserts the error, the retry count, and the
elapsed time, so no single wrong answer satisfies all three; against the
unfixed code both fail with kOperationFailed and zero retries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@nehalkpatel
nehalkpatel merged commit 7ad9640 into main Aug 27, 2026
1 check passed
@nehalkpatel
nehalkpatel deleted the fix/send-retry-loop branch August 27, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ZmqTransport::Send retry loop can never retry

1 participant