Skip to content

fix(db): prevent pool recovery lock deadlock - #224

Open
linhongyu510 wants to merge 2 commits into
apache:masterfrom
linhongyu510:fix/pool-recovery-deadlock-223
Open

fix(db): prevent pool recovery lock deadlock#224
linhongyu510 wants to merge 2 commits into
apache:masterfrom
linhongyu510:fix/pool-recovery-deadlock-223

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Aug 31, 2026

Copy link
Copy Markdown

Fixes #223.

Summary

  • Make _recover_pool() the sole owner of _recovery_lock, removing the nested acquisition of the same non-reentrant asyncio.Lock.
  • Record the observed global-pool generation before waiting for the lock, then skip recovery if another waiter has already created a newer generation.
  • Keep _recover_pool_with_lock() as a compatibility entry point that delegates to the lock-owning recovery routine.
  • Add deterministic regression coverage for missing pools, closed pools, acquisition timeouts, and concurrent recovery requests.
  • Record the user-visible reliability fix under CHANGELOG.md.

Concurrency behavior

Each recovery caller records the current pool generation before waiting. The first caller performs recovery while holding _recovery_lock and increments the generation when the new pool is installed. Later callers acquire the lock, observe the changed generation, and return without closing or rebuilding the recovered pool.

If the first recovery fails without creating a new generation, the next waiter still attempts recovery rather than incorrectly treating the failed attempt as success.

Verification

uv run pytest -q -W error
1867 passed, 85 skipped; total coverage 68.34%

uv run pytest -q -W error --no-cov test/utils/test_doris_user_pool_manager.py
21 passed

uv run ruff check doris_mcp_server/utils/db.py \
  test/utils/test_doris_user_pool_manager.py
All checks passed

uv run mypy doris_mcp_server/utils/db.py
Success: no issues found in 1 source file

uv run bandit -q -c pyproject.toml -r \
  doris_mcp_server doris_mcp_client
Passed

uv lock --check
Passed

uv build
Built sdist and wheel successfully

No live Doris instance is required for this regression because the original deadlock occurs before any database operation. The tests exercise the real _get_global_connection() paths for missing, closed, and acquisition-timeout pools with deterministic fake pools, plus concurrent direct recovery to verify one generation is created.

AI Assistance Disclosure

AI assisted with investigation, implementation review, and test drafting. The author reviewed the diff, strengthened the tests to exercise the production connection path, and ran all verification commands listed above.

Ensure the recovery routine is the sole owner of the recovery lock and skip redundant work when another waiter has already created a newer pool generation. Add regression coverage for missing and closed pools, acquisition timeouts, and concurrent recovery.

Closes apache#223
@linhongyu510
linhongyu510 marked this pull request as ready for review August 31, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global connection pool recovery deadlocks by reacquiring the same asyncio.Lock

1 participant