[Parquet] Stop byte array batches before 32 bit offsets overflow - #10912
Open
samvallad33 wants to merge 1 commit into
Open
[Parquet] Stop byte array batches before 32 bit offsets overflow#10912samvallad33 wants to merge 1 commit into
samvallad33 wants to merge 1 commit into
Conversation
A row group holding more than i32::MAX bytes of byte array values cannot be decoded into a single StringArray or BinaryArray. Today the reader fails with "index overflow decoding byte array", or panics outright on the DELTA_LENGTH_BYTE_ARRAY path. Stopping inside the values decoder does not work: by then the definition levels for those values have already been consumed by GenericColumnReader, a short values read is a hard error there, and a short read from the record reader is interpreted higher up as an exhausted column chunk. Instead ask the values decoder for a capacity bound once per data page, before any levels are decoded, and lower the record budget. A new stopped_for_capacity flag distinguishes a capacity stop from an exhausted chunk so the record reader does not spin and the array reader does not advance the page iterator. ColumnValueDecoder::values_capacity defaults to None and is called through a generic type parameter, so it folds away for every decoder that does not override it. The byte array implementation answers from a bound it already holds in the common case and only scans when the remainder of a page might not fit. Multi column projections, the predicate cache and repeated columns are not covered yet and now fail with a message that names the limit. The regression tests added in apache#9361 never reached the reader: their input array overflowed inside BinaryBuilder in the test helper, which raised the same panic message the tests asserted on. They are replaced with tests that write the row group in chunks and read it back.
samvallad33
marked this pull request as ready for review
August 30, 2026 03:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #7973.
Byte array columns break once the values in one batch exceed i32::MAX. At the default batch size of 8192 that caps the average value around 256KB. The 5MB rows in the issue only get about 409 into a batch before OffsetBuffer::try_push errors. DELTA_LENGTH_BYTE_ARRAY is worse and panics outright, because that path builds offsets with expect instead of try_push.
Three earlier attempts, #9362, #9369 and #9504, all tried to stop inside the byte array decoder and return a short count. That cannot work from there. The definition levels are already consumed by the time the values decoder runs, so there is nothing left to unwind. A short read that is not end-of-chunk gets treated as an exhausted chunk and silently drops the rest. And a decoder returning zero while the page still has data spins forever.
So the cap has to be applied before any levels are read. This adds ColumnValueDecoder::values_capacity, defaulting to None, which reports how many more values fit in the output buffer. The column reader checks it before consuming levels.
Single-column reads are correct with this. Multi-column needs a structural decision I would rather agree with you before building. Separately, the regression tests currently on main do not actually exercise the overflow.
Parts of this were written with AI assistance. I have reviewed and debugged all of it and can explain any part of it.