Skip to content

Make tranlog exit aware - #6136

Open
markhannum wants to merge 3 commits into
bloomberg:mainfrom
markhannum:tranlog-exit-aware
Open

Make tranlog exit aware#6136
markhannum wants to merge 3 commits into
bloomberg:mainfrom
markhannum:tranlog-exit-aware

Conversation

@markhannum

Copy link
Copy Markdown
Contributor

An exiting node can hang until the exit alarm fires while serving transaction-log requests from its replicants.

A tranlog request runs on an appsock-sql thread, and clean_exit() waits for those threads to exit in thrman_wait_for_all() under a 300s alarm. sql_flush_int() polls until wr_buf drains, and pollms is only cleared when the buffer empties or the write hard-errors. A replicant that breaks out of its fetch loop — sqllogfill and physrep both do, on gap-filled, generation change, or lock-desired — leaves the node mid-stream with nobody reading. The buffer never drains, poll() returns 0 every second forever, and thrman_stop_sql_connections() doesn't help because it shuts down the read side only.

Give up on a write that has made no progress for exit_flush_timeout_sec (default 10, 0 disables) while exiting. blocked_at is reset by every successful write, so a slow-but-still-draining client is unaffected and the clean-exit grace period for in-flight queries is preserved.

Also stop tranlog from handing the thread more work on the way out. The poll loop already checked db_is_exiting() but ended the scan quietly, which the client reads as an empty result and immediately re-issues. Raise an error from tranlogFilter() instead — sqlite treats the xEof return as a boolean, so an error raised from tranlogNext() by way of tranlogEof() is swallowed as an end-of-scan — and check for exit in the xNext paths, where errors do propagate.

Tests: sql_logfill_exit exits the master while replicants are actively logfilling; phys_rep_exit exits the source node a physrep is streaming from. Both require the node to reach goodbye rather than hang.

A tranlog request runs on an appsock-sql thread, and clean_exit waits
for those threads to exit in thrman_wait_for_all() under a 300s alarm.

sql_flush_int() polls until wr_buf drains.  A replicant that breaks out
of its fetch loop -- logfill and physrep both do, on gap-filled, gen
change or lock-desired -- leaves the master mid-stream with nobody
reading, so wr_buf never drains and the poll loop never ends.  Give up
on a write that has made no progress for exit_flush_timeout_sec while
exiting; blocked_at is reset by every successful write, so a peer that
is still draining is unaffected.

Also stop tranlog from handing the thread more work on the way out.
The poll loop already noticed db_is_exiting() but ended the scan
quietly, which the client reads as an empty result and retries.  Raise
an error from tranlogFilter instead -- sqlite treats the xEof return as
a boolean, so an error from tranlogNext() by way of tranlogEof() is
swallowed as an end-of-scan -- and check for exit in the xNext paths,
where errors do propagate.

Signed-off-by: Mark Hannum <mhannum@bloomberg.net>
Exit the master while replicants are actively issuing blocking
transaction-log requests, and require it to reach 'goodbye' rather than
hang until the exit alarm.

Signed-off-by: Mark Hannum <mhannum@bloomberg.net>
Physreps drive the same blocking tranlog path.  Attach a physrep, then
exit the source node it is streaming from and require it to reach
'goodbye'.

Signed-off-by: Mark Hannum <mhannum@bloomberg.net>

@roborivers roborivers left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
cdb2jdbc
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_dbname
ssl_prefer
sc_downgrade [timeout] **quarantined**

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants