fix(db): prevent pool recovery lock deadlock - #224
Open
linhongyu510 wants to merge 2 commits into
Open
Conversation
Ensure the recovery routine is the sole owner of the recovery lock and skip redundant work when another waiter has already created a newer pool generation. Add regression coverage for missing and closed pools, acquisition timeouts, and concurrent recovery. Closes apache#223
linhongyu510
marked this pull request as ready for review
August 31, 2026 12:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #223.
Summary
_recover_pool()the sole owner of_recovery_lock, removing the nested acquisition of the same non-reentrantasyncio.Lock._recover_pool_with_lock()as a compatibility entry point that delegates to the lock-owning recovery routine.CHANGELOG.md.Concurrency behavior
Each recovery caller records the current pool generation before waiting. The first caller performs recovery while holding
_recovery_lockand increments the generation when the new pool is installed. Later callers acquire the lock, observe the changed generation, and return without closing or rebuilding the recovered pool.If the first recovery fails without creating a new generation, the next waiter still attempts recovery rather than incorrectly treating the failed attempt as success.
Verification
No live Doris instance is required for this regression because the original deadlock occurs before any database operation. The tests exercise the real
_get_global_connection()paths for missing, closed, and acquisition-timeout pools with deterministic fake pools, plus concurrent direct recovery to verify one generation is created.AI Assistance Disclosure
AI assisted with investigation, implementation review, and test drafting. The author reviewed the diff, strengthened the tests to exercise the production connection path, and ran all verification commands listed above.