[Load] Isolate reused TsFile parser memory from Pipe pool - #18419
[Load] Isolate reused TsFile parser memory from Pipe pool#18419Caideyipi wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18419 +/- ##
=========================================
Coverage 43.61% 43.61%
Complexity 374 374
=========================================
Files 5393 5395 +2
Lines 385958 386039 +81
Branches 50267 50277 +10
=========================================
+ Hits 168337 168384 +47
- Misses 217621 217655 +34 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| if (newSizeInBytes == 0) { | ||
| delegate.close(); | ||
| delegate = null; |
There was a problem hiding this comment.
If resized twice with size 0, will an NPE be triggered?
There was a problem hiding this comment.
Thanks for checking. The first resize to 0 closes the delegate and sets the tracked usage to 0; a second resize to 0 returns before dereferencing the null delegate because the usage is unchanged. I added a regression assertion for the repeated zero-size resize in 277e61e.
|



Description
Problem
Load reuses TsFile insertion parsers, whose tablet, chunk, page, and modifications blocks were hard-coded to PipeDataNodeResourceManager.memory(). A large Pipe tablet backlog could therefore consume the Pipe pool and prevent Load from acquiring query memory.
Fix
Tests
Scope note
The existing Load conversion semaphore only reads Pipe capacity to size permits; this change removes parser working-memory allocations from the Pipe pool.
This PR has:
Key changed/added classes