perf(parquet): fuse consecutive filters on the same projection - #10859
Draft
haohuaijin wants to merge 7 commits into
Draft
perf(parquet): fuse consecutive filters on the same projection#10859haohuaijin wants to merge 7 commits into
haohuaijin wants to merge 7 commits into
Conversation
Contributor
Author
|
Replace the dedicated `with_same_projection_predicates` loop and the predicate-group bookkeeping in the push decoder state machine with a crate-private `FusedPredicate` that implements `ArrowPredicate`. `RowFilter::fuse_same_projection` groups consecutive predicates that share a single-leaf projection once at build time, so `ReadPlanBuilder`, `FilterInfo` and the LIMIT short-circuit work unchanged and the sync reader gets fusion for free. Within a batch the accepted rows are tracked as a `RowSelection` whose mask/selector backing follows the reader's `RowSelectionPolicy`, via the new `RowSelectionPolicy::resolve` and `RowSelection::into_boolean_buffer`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #10926, #10774, and #10776.
Rationale for this change
Consecutive same-projection predicates can repeatedly decode a column or replay it from the predicate cache. The push decoder now wraps eligible groups in a
FusedPredicateand evaluates them from one decoded stream. Predicates keep their order, and later predicates receive only surviving rows.Fusion is limited to a single top-level, non-repeated leaf. Contiguous survivors use zero-copy slices; fragmented survivors use
filter_record_batch. Selection composition usesRowSelectorruns or bitmaps according to run density, with an average-run-length threshold of 32. The synchronous reader is unchanged.What changes are included?
Performance
Measured with DataFusion
6f74e46a1, batch size 8192, and 12 partitions. Main is Arrow7d9bdfd8a; previous PR is2aee4298c; adaptive isf2981938f, before removing its sync-reader integration. The push-decoder evaluation logic is unchanged by that removal.Q25 used 114 retained samples per variant. Adaptive is 18.77% faster than main/on and approximately equal to main/off (+0.05%), but 2.33% slower than the previous PR implementation.
The synthetic matrix covers 256 three-predicate cases over 4 million Snappy-compressed rows, with the filter column projected: Int64/String, fragmented/clustered layouts, and all combinations of 99%, 50%, 20%, and 1% survivors. With 18 retained samples per case, geometric-mean latency was 1.49% lower than main/on, versus 1.68% for the previous PR. Retesting 67 cases with 84 samples per variant reduced the largest observed regression versus main/on from 10.28% in the previous PR to 3.56% in adaptive. For clustered Int64 with 99%/99%/99% survivors, the regression decreased from 10.28% to 2.03%.
This reduces some worst-case regressions, not all regressions. Fusion still trades decoding or predicate-cache replay for survivor materialization.
Testing
User-facing changes
No public API changes.