fix(parquet): skip empty data pages in add_data_page - #10930
Draft
adriangb wants to merge 1 commit into
Draft
Conversation
Content-defined chunking forces a page break at the end of every chunk except the last, without checking whether anything is buffered. When a chunk's own values have already filled the page, that forced break has nothing left to write. For a BOOLEAN column using RleValueEncoder the flush panicked with "RLE value encoder is not initialized", because the inner encoder is created lazily on the first put. For other encodings it silently wrote a data page holding zero values. Return early when no values are buffered, matching the checks already made by should_add_data_page and flush_data_pages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
GenericColumnWriter::add_data_pageflushes the encoder unconditionally. Its two neighbouring call sites already avoid calling it with an empty page (should_add_data_pagereturnsfalseat zero buffered values, anddict_fallback/flush_data_pagestestnum_buffered_values > 0), but content-defined chunking does not:Writing the chunk can already have flushed the page, when the chunk's own values reach
data_page_size_limitordata_page_row_count_limitexactly at the chunk boundary. The forced break then flushes a page with nothing buffered.For a
BOOLEANcolumn that panics.RleValueEncodercreates its inner encoder lazily on the firstputandflush_bufferdoes.take().expect("RLE value encoder is not initialized"), so flushing before anyputpanics.BOOLEANresolves toRleValueEncoderunderWriterVersion::PARQUET_2_0, or whenEncoding::RLEis set explicitly.For other encodings it does not panic but writes a data page holding zero values.
This is reachable from the public API:
WriterPropertiesBuilder::set_content_defined_chunkingplusArrowWriter.data_page_size_limitsmaller thanmax_chunk_sizeis a configuration theCdcOptions::max_chunk_sizedocs explicitly describe as supported.What changes are included in this PR?
add_data_pagereturnsOk(())whenpage_metrics.num_buffered_values == 0, making a forced page break a no-op when there is nothing to write. This brings it in line with the checks the other call sites already make.Are these changes tested?
Yes, two regression tests in
parquet/src/arrow/arrow_writer/mod.rs, both driving only the publicArrowWriter/WriterPropertiesAPI. Both fail onmain:test_arrow_writer_cdc_boolean_forced_page_break_after_flushwrites aBOOLEANcolumn with CDC and a smalldata_page_size_limit, and checks the values round-trip. Onmainit panics withRLE value encoder is not initialized.test_arrow_writer_cdc_writes_no_empty_data_pageswrites anINT32column with CDC anddata_page_row_count_limit, and asserts no data page reports zero values. Onmain121 of 611 pages are empty.Are there any user-facing changes?
Yes, both are fixes:
BOOLEANcolumn with content-defined chunking no longer panics.No public API change, and no change to files written without content-defined chunking.
This PR was written by Claude (Anthropic's AI assistant) working with @adriangb. The tests were run and the results reported above are actual.