fix(transport): make ZmqTransport::Send actually retry - #33
Merged
Conversation
The retry loop was unreachable, for two independent reasons. cppzmq's send() reports an expired ZMQ_SNDTIMEO by returning an EMPTY result and throws error_t only for everything else. Send() looked for EAGAIN exclusively inside a catch block and treated the falsy return as a hard failure, so a send timeout came back as kOperationFailed without ever reaching the loop's deadline check. Classify the attempt instead, in a TrySendOnce helper that catches the timeout in both the forms ZMQ reports it. Independently, the timeouts left no room to retry: send_timeout bounds ONE attempt and retry.total_timeout bounds all of them, and both defaulted to 1000ms, so the first EAGAIN arrived at the deadline and max_attempts and retry_delay described behaviour that could not happen. Size the default per-attempt slice to fit (3 * 300ms + 2 * 10ms <= 1000ms), and enforce the invariant at construction by shrinking send_timeout for any caller-supplied config that violates it. Clamping beats rejecting: asking for three attempts within a second is coherent, and the arithmetic that makes it fit is ours to do rather than grounds for failing startup. Two tests, both driving real backpressure by filling the send queue past ZMQ_SNDHWM, since -- contrary to what the header used to claim -- a peerless PAIR socket does not go mute. connect() creates the outbound pipe whether or not the far end is reachable and libzmq queues into it, so sends to nobody succeed until the high-water mark. Corrected those comments too. Each test asserts the error, the retry count, and the elapsed time, so no single wrong answer satisfies all three; against the unfixed code both fail with kOperationFailed and zero retries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #30.
Two causes, not one
The issue's diagnosis was right that the retry path was unreachable, but there were two independent causes — and fixing only the timeouts would not have helped.
1. The timeout was never classified as retryable. cppzmq's
send()reports an expiredZMQ_SNDTIMEOby returning an empty result, and throwserror_tonly for everything else (zmq.hpp:1967-1976).Send()looked forEAGAINexclusively inside acatchblock and treated the falsy return as a hard failure, so a send timeout came back askOperationFailedand never reached the loop's deadline check at all.2. The timeouts left no room, as reported.
send_timeoutbounds one attempt,retry.total_timeoutbounds all of them, and both defaulted to 1000ms.Changes
TrySendOnceclassifies one attempt askSent/kWouldBlock/kFailed, catching the timeout in both forms ZMQ reports it.Send()is rewritten around that, and every exhausted path now returnskTimeout— a blocked PAIR socket means the peer never took the message either way.ClampSendTimeoutToRetryBudget(), called from the constructor, enforcesmax_attempts * send_timeout + (max_attempts-1) * retry_delay <= total_timeoutby shrinkingsend_timeoutand warning. It also floorsmax_attemptsat 1, which otherwise skipped the loop entirely.send_timeout1000ms → 300ms, so the defaults satisfy the invariant unclamped: 3×300 + 2×10 = 920ms ≤ 1000ms.Clamping rather than rejecting the config: a caller asking for three attempts within a second has said something coherent, and the arithmetic that makes it fit is ours to do rather than grounds for failing startup.
A correction to the issue, and to the header
A peerless PAIR socket does not go mute, contrary to the issue's closing note and the header's own comments.
connect()creates the outbound pipe whether or not the far end is reachable, and libzmq queues into it, so sends to nobody succeed untilZMQ_SNDHWM(1000) messages are outstanding. Measured directly. The stale comments are corrected here.That is also why the tests below fill the queue rather than simply pointing at an unbound endpoint — the obvious setup would have passed against the broken code.
Tests
Two tests, each asserting the error, the retry count, and the elapsed time, so no single wrong answer satisfies all three:
SendAttemptsEveryRetryWhenSocketBlocks— measured 326ms against a predicted 3×100 + 2×10 = 320ms.SendRetriesWhenSendTimeoutClaimsTheWholeBudget— the shape of the config that shipped. Measured 605ms against a predicted 3×193 + 2×10 = 599ms, bounded both ways so it distinguishes shortened attempts from full-length ones.Against the unfixed code both fail, with
kOperationFailedand zero retries.Verification
cmake --workflow --preset=host-debugand--preset=host-releaseboth green — 35/35 tests, including the Python integration suite.tools/format.sh --checkclean; clang-tidy clean via the build.🤖 Generated with Claude Code