Core, Spark: Introduce shared MultiColumnTerm to de-duplicate Z-order and Hilbert clustering - #17893
Open
GGraziadei wants to merge 1 commit into
Open
Core, Spark: Introduce shared MultiColumnTerm to de-duplicate Z-order and Hilbert clustering#17893GGraziadei wants to merge 1 commit into
GGraziadei wants to merge 1 commit into
Conversation
… and Hilbert clustering
Member
Author
|
Please consider GGraziadei#1 as the natural follow-up once this PR is merged. The idea is to extend the grammar to natively support transformations, improving the quality of Z-order and Hilbert clustering. E.g. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17657.
Follow-up to #16827, which deliberately mirrored the Z-order surface to keep that diff reviewable. This PR is the deferred de-duplication, along the lines @RussellSpitzer suggested in review ("we'll probably want a
MultiColumnTermor something like that in the future").What changed
Core — new
org.apache.iceberg.expressions.MultiColumnTerm, an immutableTermover an ordered list of column references.ZorderandHilbert(previously identical apart from the class name) become two-line subclasses. Public constructors andrefs()are unchanged; the parsed SQL grammar is untouched.Spark 4.1 —
SparkZOrderFileRewriteRunnerandSparkHilbertFileRewriteRunnermove onto a new abstractSparkCurveFileRewriteRunner, parameterised by the internal value column name, the exact per-curve error messages, and the combine step. The shared pieces — internal column schema/sort order, column validation (validZOrderColNames/validHilbertColNameswere byte-for-byte the same logic),sortSchema,sortedDF, and the per-column ordered-bytes conversion — now live in one place. Each runner keeps only what is genuinely curve-specific: Z-order its two options and the interleave combine (~120 lines), Hilbert its fixed per-column bit width and the Hilbert-index combine (~75 lines). Adding a further curve is now a small, local change.Dispatch — the duplicated
zorder/hilbertbranches inSpark3Util.toIcebergTermshare one reference-extraction helper, andRewriteDataFilesProcedurecollapses its two per-curve term lists into a singleList<MultiColumnTerm>(mixing curves is detected by distinct term classes; error messages unchanged).What did not change
Engine-side behaviour is bit-for-bit identical, which the existing tests enforce: every pre-existing Z-order and Hilbert test passes unmodified (
TestSparkFileRewriteRunners, the Z-order/Hilbert cases ofTestRewriteDataFilesAction,TestRewriteDataFilesProcedure). The only test change is the newTestMultiColumnTermin core. All user-facing error messages are preserved exactly. Spark 3.5/4.0 are untouched.Out of scope
The wider "arbitrary function via
Expressions" refactor discussed in #16827 review builds naturally on this abstraction but is split out per the issue, to keep this diff mechanical and easy to verify.