Skip to content

[client] Expose Arrow changelog batches - #4006

Open
jordepic wants to merge 1 commit into
apache:mainfrom
jordepic:client-arrow-changelog-batches
Open

[client] Expose Arrow changelog batches#4006
jordepic wants to merge 1 commit into
apache:mainfrom
jordepic:client-arrow-changelog-batches

Conversation

@jordepic

@jordepic jordepic commented Aug 15, 2026

Copy link
Copy Markdown

Generated-by: Codex (GPT-5) following the guidelines

Purpose

Linked issue: close #4005

Expose the Arrow representation that Fluss already stores in ARROW primary-key log batches through the Java client, including the per-row changelog sidecar.

StreamFusion needs this path for its Flink Fluss source so live-log batches can remain Arrow-backed instead of being materialized as RowData first. Snapshot records can still use the normal conversion path initially; this PR provides the direct Arrow path for the unbounded changelog.

Brief change log

  • Promote LogScanner.pollRecordBatch and its result types to public evolving APIs.
  • Decode and retain the stored change-type vector for primary-key Arrow batches.
  • Support existing server-side non-empty top-level projection pushdown on the Arrow batch path.
  • Expose per-row ChangeType access and a read-only view of the raw sidecar.
  • Keep Arrow rows and change types aligned across projection, offset slicing, and truncation.
  • Use a default unsupported scanner implementation to preserve compatibility for existing LogScanner implementations.
  • Remove the internal implementation cast from the Flink tiering reader.

Tests

  • Added LogScannerITCase#testPollArrowBatchesWithPrimaryKeyChangelog covering insert, update-before, update-after, delete, projected Arrow columns, and mid-batch offset slicing.
  • Ran the new test together with testPollArrowBatchesWithSchemaEvolution: 2 tests passed.
  • Built fluss-flink-common and its dependencies with test compilation enabled: success.
  • Spotless and Checkstyle passed for affected modules as part of the Maven builds.

API and Format

This adds public-evolving Java client APIs for Arrow batch polling and changelog access. Existing LogScanner implementations remain compatible through a default method.

There is no storage-format change: the implementation exposes the change-type vector already stored before the Arrow IPC payload in primary-key log batches and reuses the existing server-side Arrow projection path.

Caller-provided Arrow allocator support is intentionally left for follow-up because allocator ownership and lifecycle need a separate API decision.

Documentation

Public API Javadocs describe format restrictions, ownership, close requirements, and changelog behavior. No user-facing storage or configuration documentation changes are required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[client] Expose Arrow changelog batches for primary-key tables

2 participants