[Maven4] Migrate from Maven Archiver to standard jar tool - #508
[Maven4] Migrate from Maven Archiver to standard jar tool#508desruisseaux wants to merge 20 commits into
jar tool#508Conversation
Update all m4/pom.xml to use maven-jar-plugin 4.0.0-beta-2-PR508-SNAPSHOT which provides automatic JAR-per-module creation. Additional changes: - Convert example_addExports_manifest/m4/src/modmain from symlink to real directory to support MANIFEST.MF with Add-Exports and Main-Class - Add Maven 4 Migration Notes to example_addExports_manifest/README.adoc - Document resource handling in README.adoc Dependencies: - apache/maven-jar-plugin#508: Automatic JAR-per-module creation - apache/maven#11505: Module-aware resource copying 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Introduced in the course of support-and-care/maven-support-and-care#137
f564cdc to
21db1f0
Compare
5a235c8 to
5886fef
Compare
|
Thanks for this migration, @desruisseaux — moving JAR creation to the Reviewing it, I dug into the CI failures, and they share a root theme: the multi-release/module handling in They're on
Because these are order/vendor-dependent they only surface on CI (Linux+Zulu); the |
…ard `java.util.spi.ToolProvider` API (requires Java 9+). Files in the `classes` directories are dispatched to the `--manifest` and `--release` options, which allow additional verifications by the `jar` tool. Derive a POM for each individual JAR file as the intersection of the project model and the module-info of the JAR file. Side effects: * Remove the default `**/package.html` exclude. * Automatic use of `META-INF/MANIFEST.MF` file found in `classes` directory. * Automatic Multi-Release always enabled, unless `detectMultiReleaseJar` is set to `false`. * In a multi-module project, use Java module names as artifact names. * Whether the `--date` option is supported depends on the Java version. * Re-run the `jar` tool with the --validate operation mode if validation was not implicit.
5886fef to
046fbb2
Compare
|
Thanks! I cherry-picked the commits with modifications:
|
3ea1283 to
bb93fec
Compare
The plugin runs `jar --validate` after creating each archive. With the `jar` tool bundled in JDK 17 and 18, that validation crashes on any class compiled as a record (JDK-8282446, "This feature requires ASM8"). The issue was fixed in JDK 19 via the ASM 9.2 upgrade (JDK-8282508), but the fix was not backported to JDK 17u/18u. Since Maven 4 runs on JDK 17+, packaging a module that contains a record may fail in project with default configuration if using a JDK older than 19. Guard the validation pass on Runtime.version().feature() >= 19 and log an INFO line naming the skipped archive. The archive was already created by the `--create` pass. Behaviour on JDK 19+ is unchanged. Add an IT which tests a modular JAR containing a record and expects the build to succeed, so it turns the jdk-17 CI leg red until the validation pass is guarded for JDK 17/18. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
`Archive` seeded its base (version-less) `FileSet` from whichever directory first created the `Archive` via `computeIfAbsent(…)` in `FileCollector.enterModuleDirectory(…)`. Directory iteration order is unspecified, so when the walk enters "META-INF/versions-modular/<n>/<module>" before the base "<module>" dir, the base `FileSet` was bound to a version-specific directory with no module-info.class. The problem was not really the version-specific directory, but its association to the `null` key in `filesetForRelease`, which makes it the first map entry. Consequently `baseRelease().directory` pointed there, `ModuleFinder.of(...)` found no module, and `PomDerivation` dereferenced a null `ModuleReference` -> NPE. The "green on JDK 19" was accidental walk order, not a version gate. Add a `version` argument to the `Archive` constructor, which keep the `null` key free for the real base version when the walk will reach that version. Bug description partially from Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
`setMainClass(…)` unconditionally removes the `Main-Class` attribute from the plugin manifest (to either pass it via `--main-class` instead or to ignore it), but that manifest is shared across every module of a module hierarchy. The first module processed consumed the attribute, so when a non-owning module was processed first (directory iteration order is unspecified), the owning module never received its main class. Give each module a copy of the shared manifest in `ToolExecutor.writeSingleJAR` so processing order no longer decides the outcome. Add `ArchiveTest` with deterministic unit tests for this and for the base-release binding (both previously exercised only by ITs whose result depends on filesystem walk order). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
bb93fec to
0238e7d
Compare
`FileSet.add(…)` relativized only the first file added and stored the rest as absolute paths. This heuristic rule is based on the observation that is seems to be a `jar` tool requirement. However, which file is first depends on the unspecified directory iteration order. On some platforms, the tests fail with "names do not match" error message. This make the first file deterministic by choosing the shortest one. Issue identified by: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
I agree, and this is one reason why I disagree with the importance given to strict (as opposed to semantically) reproducible builds. I think that it can even deserve the real goal, which is security. If some Java tools produce vulnerable code (e.g., Javadoc HTML frame injection vulnerability in 2013), it is difficult to know if a JAR file has been produced by patched tools if we don't have a useful Regarding the split of metadata generation (
Regarding the JAR file validation, this is one of the reasons for this migration to the |
But is is worth the effort? It would be a hack needed for JDK 17 and 18 only. And since the file ordering issue (JDK-8276764) was fixed in JDK 18 only, even if we fix the timestamp the build would still non-reproducible on JDK 17. Therefore, setting the timestamp would be useful for only one specific JDK version: 18. |
|
Hello Martin,
Bug fixes/enhancements in the JDK are sometimes backported. Those backports are tracked/linked against the original issue. For this specific issue https://bugs.openjdk.org/browse/JDK-8276764 , under the issue description, there's a backports section which shows that this enhancement was backported to JDK 17 (the Resolved/Fixed status on those backports is a sign that the work is complete). Having said that, my comment is mostly informational and isn't meant as an input to whether or not some post processing should be done for the JAR files. |
|
IMO it's critical to support binary reproducibility for all JDKs since minimal 17, otherwise it'll be a regression. |
|
Thanks @jaikiran for the clarification. Then indeed, editing timestamps in a post-processing would work on both JDK 17 and 18. Whether we should do this post-processing is still uncertain to me. Not doing this post-processing would be a regression only for users wanting binary reproducible builds with the tools of JDK 17 or 18. Users who do not upgrade to a more recent Java version at least for building (even if they target a lower release with |
stating that for maven 4 java 17 bytecode is supported using a java 21 or more (to speak "LTS") doesn't sound crazy to me
ZipFileSystem#sync can be worth benching on a big jar (some thousands of classes), ideally it is very small compared to producing the zip (jar) itself but worth evaluating before thinking this is an option IMHO - since we know we can always do it right directly if needed and JDK version is a blocker (once again not for me) |
Instead of enumerating all included files, move excluded files in a temporary directory then add the original directory as a whole. The intend is to let the `jar` plugin generates directory entries.
gnodet
left a comment
There was a problem hiding this comment.
Review: [Maven4] Migrate from Maven Archiver to standard jar tool
Thank you for this substantial and well-motivated refactoring, @desruisseaux! The migration to the JDK's built-in jar tool via ToolProvider brings genuine technical benefits — safer multi-release JAR validation (#484), proper module hierarchy support, excellent debuggability with target/jar.args, and a foundation to leverage future jar tool improvements.
The overall code quality is high, with thoughtful Javadoc, defensive programming, and careful attention to ordering-sensitive edge cases (the ArchiveTest tests for iteration-order independence are excellent).
Highlights
jar --releasevalidation catches multi-release API incompatibilities that the old Plexus approach missed entirelytarget/jar.argsdebug file with platform-aware CLI tips is a first-class debugging feature- Copy-on-write manifest handling correctly prevents the shared manifest from being consumed by the first module
- Strong integration test coverage for new scenarios (
multi-module,multirelease-with-modules,records-jar-validate)
Summary of Findings
| Severity | Count | Key Items |
|---|---|---|
| Blocking | 1 | validate() dead-code condition (B1) |
| Important | 5 | NPE risk in PomDerivation (I1), resource leak in MetadataFiles.close() (I2), cross-device link failure (I3), NumberFormatException on "-" (I4), silent reproducible build degradation (I5) |
| Suggestion | 6 | Unit test gaps for FileCollector/ToolExecutor (S1-S2), visitFileFailed() override (S3), dependency cleanup (S4-S5), missing IT for <jarArgs> (S6) |
Upstream Dependencies Note
The PR depends on unmerged PRs (maven#11425, maven#11549, maven-compiler-plugin#998). This is understood from the PR description and not a code issue per se, but the integration timeline should be tracked.
This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
For consistency with integration test added in previous commit.
…s is requested. Before this commit, the "Created-By" value generated by the `jar` tool was overwritten only if `addDefaultEntries` was `true`. Reported by Sergey Chernov on apache#508. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
|
@seregamorph, @hboutemy: I'm almost done in fixing the issue raised last week, with the help of @ascheman commits (thanks!). I have made one change which I suspect will be highly controversial. If strictly reproducible builds has been requested, the JDK version in the |
|
I would remove "on unspecified JDK". I might add the actual JDK version since reproducible builds require the same toolchain for now. For the record, I do not think "semantically reproducible builds" are a thing. Byte-per-byte comparison is the only thing that works. |
Are we requiring toolchain? I thought that we were betting on different JDKs producing the same bytecode when used with the same
For the security goal, I have not yet seen a technical argument explaining why semantically reproducible builds would not work. On the contrary, I think that it is more secure than strictly reproducible builds for the reason given above. In addition of security goal, it seems that there is also a caching goal. It has been argued that Docker images use hash-code for caching JAR files. This is a more difficult case and may be a reason to keep strictly reproducible builds as an option, but not necessarily a reason for makes it the default if semantically reproducible builds can be guaranteed. |
…s is requested. Before this commit, the "Created-By" value generated by the `jar` tool was overwritten only if `addDefaultEntries` was `true`. Reported by Sergey Chernov on apache#508. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Martin Desruisseaux <martin.desruisseaux@geomatys.com>
4bd866b to
3e0852e
Compare
|
Ok, I removed the "on unspecified JDK" part. I think that the risk of controversy is almost certain. |
|
A different compiler version can easily choose to perform different optimizations; e.g. to unroll a loop or not, while remaining fully spec conformant. This produces different byte code, even when compiling for the same Java version. I'm not sure whether determining that two different versions of the byte code for a method produce the same output for the same inputs is equivalent to solving the halting problem, but it's certainly non-trivial and not something I want to rely on for reproducible builds. |
Yes, this is exactly why I think that strictly reproducible builds currently works only by chance. Doing semantically reproducible builds at the bytecode level would be challenging, but if we start with only the easy part (ignoring timestamp, compression and a few selected metadata such as |
…s supported. It allows the creation of quasi-reproducible JAR files on JDK 17/18.
| p -> p.startsWith("META-INF" + File.separatorChar + "versions" + File.separatorChar))) { | ||
| getLog().debug("Adding 'Multi-Release: true' manifest entry."); | ||
| archive.addManifestEntry(Attributes.Name.MULTI_RELEASE.toString(), "true"); | ||
| protected String getOutputTimestamp() { |
There was a problem hiding this comment.
why copy/paste https://github.com/apache/maven-archiver/blob/master/src/main/java/org/apache/maven/shared/archiver/MavenArchiver.java#L691
instead of just calling the existing method?
There was a problem hiding this comment.
Because it returns a String instead of an Instant. The String is the type of argument expected by the jar tool. By passing the string without going through a round of "parsing as an Instant then reformat as a String" (except for the case of number of seconds), we let the jar tool decide which syntax it accepts.
Another reason is that building the MANIFEST.MF content is the only MavenArchiver service that the plugin is still using, and that service could be take in charge by the plugin itself in a future commit. In such case, the dependency to Maven Archiver could be removed.
gnodet
left a comment
There was a problem hiding this comment.
Re-Review — Updated PR State (commit aeb815c)
Thank you @desruisseaux for the thorough work addressing the prior review findings. Here's an updated assessment.
Previous Findings — Status
| Prior Finding | Status | Notes |
|---|---|---|
B1 — validate() dead code |
✅ Fixed | Changed to keySet().stream().anyMatch(Objects::nonNull) — correct |
I1 — PomDerivation NPE |
toRealPath() added, but one NPE risk remains (see inline comment) |
|
I2 — MetadataFiles.close() temp leak |
✅ By design | Documented as consistent with cancelFileDeletion() behavior |
I3 — linkOrCopy cross-device |
✅ Fixed | Catches both UnsupportedOperationException and IOException |
I4 — NumberFormatException on "-" |
✅ Fixed | Try-catch wrapping into clear MojoException |
| I5 — Reproducible build JDK 17/18 | ✅ Redesigned | JDK check moved to ToolExecutor; partial reproducibility with clear warning — good compromise |
New Findings
Two new findings are posted as inline comments:
- [Important]
ExcludedFiles.move()— Moving a directory over a temp file may fail on POSIX (ENOTDIRfromrename(2)) - [Important]
PomDerivationline 170 —fromURI.get(root.toUri())can return null → NPE before the null check ondescriptor
Positive Observations
- Directory entries are now preserved — The
ExcludedFilesapproach solves a real problem for Spring Boot@ComponentScancompatibility. - Reproducible
Created-By— The newcreatedBy()method produces a JDK-independent value, fixing byte-for-byte reproducibility across environments. - Thorough integration tests — The four new ITs (
directory-entries/,directory-entries-mr/,manifest-created-by/,multimodule-multirelease-with-excludes/) cover real-world scenarios well. - Parent-child deduplication in
Archive.arguments()is correct —SimpleFileVisitorguarantees depth-first order.
Overall the PR is in excellent shape. The two inline findings are straightforward to address.
gnodet
left a comment
There was a problem hiding this comment.
Re-Review — Updated PR State (commit aeb815c)
Thank you @desruisseaux for the thorough work addressing the prior review findings. Here's an updated assessment.
Previous Findings — Status
| Prior Finding | Status | Notes |
|---|---|---|
B1 — validate() dead code |
✅ Fixed | Changed to keySet().stream().anyMatch(Objects::nonNull) — correct |
I1 — PomDerivation NPE |
toRealPath() added, but one NPE risk remains (see inline comment) |
|
I2 — MetadataFiles.close() temp leak |
✅ By design | Documented as consistent with cancelFileDeletion() behavior |
I3 — linkOrCopy cross-device |
✅ Fixed | Catches both UnsupportedOperationException and IOException |
I4 — NumberFormatException on "-" |
✅ Fixed | Try-catch wrapping into clear MojoException |
| I5 — Reproducible build JDK 17/18 | ✅ Redesigned | JDK check moved to ToolExecutor; partial reproducibility with clear warning — good compromise |
New Findings
Two new findings are posted as inline comments:
- [Important]
ExcludedFiles.move()— Moving a directory over a temp file may fail on POSIX (ENOTDIRfromrename(2)) - [Important]
PomDerivationline 170 —fromURI.get(root.toUri())can return null → NPE before the null check ondescriptor
Positive Observations
- Directory entries are now preserved — The
ExcludedFilesapproach solves a real problem for Spring Boot@ComponentScancompatibility. - Reproducible
Created-By— The newcreatedBy()method produces a JDK-independent value, fixing byte-for-byte reproducibility across environments. - Thorough integration tests — The four new ITs (
directory-entries/,directory-entries-mr/,manifest-created-by/,multimodule-multirelease-with-excludes/) cover real-world scenarios well. - Parent-child deduplication in
Archive.arguments()is correct —SimpleFileVisitorguarantees depth-first order.
Overall the PR is in excellent shape. The two inline findings are straightforward to address.
`createTemporaryDirectory` instead of `createTemporaryFile`. Also fix a wrong NullPointerException in `PomDerivation`. These changes adress issues reported on the pull request.
This is a major refactoring of the Maven JAR Plugin for using the standard
jartool instead of Maven Archiver. The standard tool is available through thejava.util.spi.ToolProviderinterface, which was introduced in Java 9. Since Maven 4 upgraded its requirement from Java 8 to Java 17, the use of that interface is no longer problematic.Rational
In early Java days, the
jartool was equivalent to azipcommand with a different syntax and a little bit of special processing for theMETA-INF/MANIFEST.MFfile. Because of this quasi-equivalence, it was not difficult to create JAR files ourselves using any library capable to write ZIP files. This is what Maven Archiver does, together with supporting other archive formats. But today, thejartool became more sophisticated. It now includes options for verifying the consistency of multi-release JAR files, options for updatingmodule-info.class, provides security features specific to Java, etc.. This evolution can be seen in the "Modular JAR files" section of the Maven Plugin documentation, which states that the plugin uses thejartool for updating the JAR file in a way that the Maven Archiver can't do easily. Therefore, since Maven 4 requires Java 17 and since that Java version gives us an easy access to thejartool through thejava.util.spi.ToolProviderinterface, it may be time to abandon our manual creation of a JAR file and rely fully on thejartool instead.Benefits
Safer multi-release JAR files
In a multi-release JAR file, blindingly storing the content of
META-INF/versions/directories as if they were ordinary resources is not equivalent to using thejar --releaseoption. The difference is that in the latter case, thejartool performs some consistency checks. This issue was independently reported by user in #484, which is fixed by this pull request. Note that if this verification is not desired, Maven users can disable it by setting thedetectMultiReleaseJarplugin option tofalse.Main class managed by the
jartoolIn a modular JAR file, it is no longer sufficient to declare the main class in the
Main-Classentry of theMANIFEST.MFfile. The main class needs to be specified by the--main-classoption of thejartool, which will updatemodule-info.class. For compatibility reason the Maven JAR plugin gets the option value from that manifest entry, but internally the use of thejartool is mandatory.Options not yet supported by the plugin
This approach allows to add a
<jarArgs>configuration option, similar to the<compilerArgs>in the Maven Compiler Plugin. Such option would allow developers to use new tool arguments before they are supported by the plugin.Examples of options not yet supported by the plugin are
--module-versionand--hash-modules. Explicit support for those options could be added in a future plugin version, especially since security is becoming more and more a concern.Easier debugging and toolchain
This approach makes easy to generate a
target/jar.argsfile when the build fails or when Maven is run in verbose mode. This is similar to the compiler plugin generating atarget/javac.argsfile. This file allows the user to test easily on the command-line, which makes debugging faster. Likewise, the options can also be passed to another tool, which makes easier to resolve #439 as well.Behavioral changes
The plugin behaviour after the proposed refactoring is different than version 3 in the following aspects:
Removal of default
**/package.htmlexcludesThe current plugin version uses an undocumented
**/package.htmldefault excludes. This default seems to exist since the initial revision in March 2004, but I saw no explanation for this oddity. This default is not mentioned in the documentation. The removal of this oddity is necessary for allowing the proposed new plugin implementation to specify only some root directories to thejartool, since the tool can traverse the directory tree itself.Automatic use of
MANIFEST.MFAfter this refactor, the plugin automatically uses the
META-INF/MANIFEST.MFfile found in theclassesdirectory. Before this refactor, the plugin used that file only if explicitly specified in the<manifestFile>archive configuration. The previous policy was discussed in #255. The new policy is a natural consequence of the way that the JAR plugin is reimplemented, and also more useful in the context of multi-module (in Java module sense) projects since each module could contain its ownMANIFEST.MFfile.Multi-module projects
The refactoral support multi-module projects, including projects that are both multi-module and multi-release, as discussed in apache/maven-compiler-plugin#998.
Dependencies
This pull request depends on a Maven release which include the following pull requests:
META-INF/versionswhen the sub-directories are multi-module maven-compiler-plugin#998