Make tranlog exit aware - #6136
Open
markhannum wants to merge 3 commits into
Open
Conversation
A tranlog request runs on an appsock-sql thread, and clean_exit waits for those threads to exit in thrman_wait_for_all() under a 300s alarm. sql_flush_int() polls until wr_buf drains. A replicant that breaks out of its fetch loop -- logfill and physrep both do, on gap-filled, gen change or lock-desired -- leaves the master mid-stream with nobody reading, so wr_buf never drains and the poll loop never ends. Give up on a write that has made no progress for exit_flush_timeout_sec while exiting; blocked_at is reset by every successful write, so a peer that is still draining is unaffected. Also stop tranlog from handing the thread more work on the way out. The poll loop already noticed db_is_exiting() but ended the scan quietly, which the client reads as an empty result and retries. Raise an error from tranlogFilter instead -- sqlite treats the xEof return as a boolean, so an error from tranlogNext() by way of tranlogEof() is swallowed as an end-of-scan -- and check for exit in the xNext paths, where errors do propagate. Signed-off-by: Mark Hannum <mhannum@bloomberg.net>
Exit the master while replicants are actively issuing blocking transaction-log requests, and require it to reach 'goodbye' rather than hang until the exit alarm. Signed-off-by: Mark Hannum <mhannum@bloomberg.net>
Physreps drive the same blocking tranlog path. Attach a physrep, then exit the source node it is streaming from and require it to reach 'goodbye'. Signed-off-by: Mark Hannum <mhannum@bloomberg.net>
roborivers
suggested changes
Aug 13, 2026
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
cdb2jdbc
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_dbname
ssl_prefer
sc_downgrade [timeout] **quarantined**
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An exiting node can hang until the exit alarm fires while serving transaction-log requests from its replicants.
A tranlog request runs on an appsock-sql thread, and
clean_exit()waits for those threads to exit inthrman_wait_for_all()under a 300s alarm.sql_flush_int()polls untilwr_bufdrains, andpollmsis only cleared when the buffer empties or the write hard-errors. A replicant that breaks out of its fetch loop — sqllogfill and physrep both do, on gap-filled, generation change, or lock-desired — leaves the node mid-stream with nobody reading. The buffer never drains,poll()returns 0 every second forever, andthrman_stop_sql_connections()doesn't help because it shuts down the read side only.Give up on a write that has made no progress for
exit_flush_timeout_sec(default 10, 0 disables) while exiting.blocked_atis reset by every successful write, so a slow-but-still-draining client is unaffected and the clean-exit grace period for in-flight queries is preserved.Also stop tranlog from handing the thread more work on the way out. The poll loop already checked
db_is_exiting()but ended the scan quietly, which the client reads as an empty result and immediately re-issues. Raise an error fromtranlogFilter()instead — sqlite treats thexEofreturn as a boolean, so an error raised fromtranlogNext()by way oftranlogEof()is swallowed as an end-of-scan — and check for exit in thexNextpaths, where errors do propagate.Tests:
sql_logfill_exitexits the master while replicants are actively logfilling;phys_rep_exitexits the source node a physrep is streaming from. Both require the node to reachgoodbyerather than hang.