[flink] Fix ClassNotFoundException for RoaringBitmap in shaded Flink connector JAR

[pbanakar]

Purpose

Linked issue: close #3980

Fix ClassNotFoundException: org.roaringbitmap.RoaringBitmap at runtime when using FlussCatalog bitmap functions introduced in FIP-37 (rb_build_agg, rb_cardinality, rb_or_agg).

Brief change log

Root cause: RoaringBitmap is declared as compile scope in fluss-flink-common/pom.xml, but Maven's dependency mediation resolves it as test scope in the version-specific Flink connector modules. This happens because fluss-flink-common:test-jar is a test-scoped dependency in those modules, and its transitive RoaringBitmap dependency overrides the compile scope from the main artifact. As a result, the maven-shade-plugin excludes org/roaringbitmap/* classes from the final shaded connector JAR.

Confirmed by: ./mvnw dependency:tree -pl fluss-flink/fluss-flink-1.20 | grep roaring
→ org.roaringbitmap:RoaringBitmap:jar:1.3.0:test ← wrong scope

jar tf fluss-flink-1.20-*.jar | grep "org/roaringbitmap" → (empty) ← classes missing from JAR

Fix: Add an explicit compile-scope RoaringBitmap dependency in fluss-flink-1.18, fluss-flink-1.19, fluss-flink-1.20, and fluss-flink-2.2 pom.xml files to override the test scope resolution.

After fix: jar tf fluss-flink-1.20-*.jar | grep "org/roaringbitmap" → org/roaringbitmap/RoaringBitmap.class (and all other classes present)

Tests

Verified end-to-end locally on Flink 1.20:

• Built apache/fluss-quickstart-flink from main branch with fix applied
• Ran full user profile quickstart using rb_build_agg and rb_cardinality
• Confirmed correct results — unique_visitor_count and total_clicks accumulating in real time with no errors

API and Format

No API or storage format changes. This is a packaging fix only — 4 pom.xml files modified.

Documentation

No documentation changes. This fix unblocks the Real-Time User Profile quickstart tutorial (PR #2669 ) which demonstrates these functions.

[platinumhamburg]

@pbanakar Thanks for investigating and fixing this runtime packaging issue.

At first glance, adding a third-party dependency to the shade configuration of all four version-specific connector modules looks unusual, since these modules previously only assembled Fluss artifacts. After reviewing the dependency and packaging structure again, however, I think handling RoaringBitmap at the final connector level is reasonable in this case.

The version-specific modules exclude all transitive dependencies of fluss-flink-common, so RoaringBitmap must be declared explicitly before it can be included in the final connector JAR. Moving the shade step into fluss-flink-common would instead change that library's published artifact and create inconsistencies between its main JAR, test JAR, sources, and published POM. Introducing a separate shaded module also seems unnecessarily heavy for this fix.

There are two remaining issues:

1. Please relocate org.roaringbitmap to org.apache.fluss.shaded.org.roaringbitmap, consistent with the server-side packaging, to avoid potential dependency conflicts.

2. Please add the corresponding NOTICE file to each version-specific connector module. This should also resolve the current License Check failures.

Since this is the first time these version-specific connector modules explicitly include a third-party artifact in their own shade configuration, it would be helpful to get another opinion on this dependency boundary. @polyzos @wuchong , could you please take a look as well?

[pbanakar]

Hi @platinumhamburg, Thank you for your support, addressed both issues, PTAL!

• Added relocation org.roaringbitmap → org.apache.fluss.shaded.org.roaringbitmap consistent with fluss-server packaging
• Added NOTICE files to all 4 Flink connector modules listing RoaringBitmap:1.3.0

[Jackeyzhe]

Thanks for the fix — verified the approach works: RoaringBitmap 1.3.0 has no transitive dependencies (the old org.roaringbitmap:shims split was dropped in 1.x), so a single <include> covers everything, and the base-path classes are correctly relocated.

One packaging nuance worth noting (hit the same class of issue in #3884): RoaringBitmap 1.3.0 ships as a multi-release JAR — it contains META-INF/versions/11/org/roaringbitmap/ArraysShim.class alongside the base org/roaringbitmap/ArraysShim.class. The shade plugin's relocation doesn't apply to META-INF/versions/** entries, so the versions/11 copy will land in the connector uber-jars unshaded, under the original org.roaringbitmap package.

This is functionally harmless — all references are relocated to the shaded path, and the base-path ArraysShim (the pre-Java-11 fallback) is included and relocated, so runtime behavior is correct. But for packaging hygiene, it may be worth excluding the MRJ copies so no unshaded third-party classes remain in the jar, e.g. with a filter on the artifact:

<filter>
    <artifact>org.roaringbitmap:RoaringBitmap</artifact>
    <excludes>
        <exclude>META-INF/versions/**</exclude>
    </excludes>
</filter>

(Excluding them is safe: the base-path class is the compatible fallback; only the Java-11-optimized variant is dropped.)

Aside from that, +1 from me — and I also agree with @platinumhamburg's take on the dependency boundary: shading at the connector level keeps fluss-flink-common's published artifacts unchanged, which seems like the right call here.