Accept Java module names as attached artifactId even if they differ from the project's artifactId

[desruisseaux]

Pull request #1838 added a verification in DefaultProjectManager making sure that an artifact attached to a project has the same groupId, artifactId and version as the project. This pull request relaxes that restriction by allowing the artifactId to be different, but still requires the groupId and version to be the same.

The intend is to support multi-module projects where more than one artifact may be produced. It was briefly discussed in apache/maven-compiler-plugin#959 (comment). This is needed by the new Maven JAR Plugin proposal. Note that classical projects (package hierarchy) are expected to continue to use the same artifactId as before.

[desruisseaux]

Hello @gnodet. I have a question related to this pull request. Is it possible to edit the consumer POM of the attached artifacts for:

• Setting the value of the <artifactId> element to the artifact.getArtifactId() value.
• Filter the dependencies for keeping only the ones that are declared in the module-info file.
• Change the <optional> value of some dependencies depending on whether the dependency is declared as static in the module-info file.

Any tip would be greatly appreciated.

[kwin]

If the GAV is no longer the same how can the attached artifact share the same POM (with a different artifactId)?

[desruisseaux]

If the GAV is no longer the same how can the attached artifact share the same POM (with a different artifactId)?

A consumer POM needs to be generated for each attached artifact. This is needed anyway, because the dependencies are no longer the dependencies declared in the POM. They are the intersection of the dependencies in the POM and the requires statements in the module-info. Note that the latter was true even before all this module hierarchy work. It was just not clearly realized by all users and largely ignored by Maven.

[hboutemy]

If the GAV is no longer the same how can the attached artifact share the same POM (with a different artifactId)?

I have the same concern, what is the pom of "automatic sub-artifactId Java Module" vs the base artifactId?

this is really key, ad requires a clear design before trying to implement

[hboutemy]

small note on label: "backport-4.0.x" is for backport from master branch (currently 4.1.x) to 4.0.x branch: this PR is not a backport; AFAIK

[ascheman]

If the GAV is no longer the same how can the attached artifact share the same POM (with a different artifactId)?

I have the same concern, what is the pom of "automatic sub-artifactId Java Module" vs the base artifactId?

this is really key, ad requires a clear design before trying to implement

The consumer POM for a Java module JAR would have a different artifactId (derived from the module name).
In the context of attachArtifact (method where the check is performed) it would be possible to decide whether it's a multi-module project (or single-module).
Relaxation of the artifactId check would only be necessary if its multi-module. For single-module projects the artifactId should perhaps not change (-> no relaxation)?

[hboutemy]

IIUC, you expect to use the same consumer pom for automatic artifactIds of Java module JAR, just update the artifactId?
and implicitely no build pom?

for better understanding of this plan (that would deserve more than a small discussion in an issue in Maven core: is there a doc somewhere?), is there a convention between original artifactId and automatic artifactIds?

and is there a jar associated to the original artifactId?

first thought: instead of dropping the check, I'd clarify the creation an auto-artifactId from a base artifactId, as I think it is more a new pattern to enable, instead of a control to disable
for example, when I read "This is needed by the new Maven JAR Plugin proposal apache/maven-jar-plugin#508.", I don't get the relationship with using jar vs MavenArchiver: we need to be cautious

[ascheman]

IIUC, you expect to use the same consumer pom for automatic artifactIds of Java module JAR, just update the artifactId? and implicitely no build pom?

Well, the overall context is a project where you have multiple Java modules. I have provided some examples here.

If I pick one (cf. an arbitrary pom.xml, same holds for almost each example, cf. their m4 subdirectories), there is usually just one build POM. But the build would result in multiple binary artefacts (or it could result in such as this is work-in-progress), one for each module.

For each such module JAR there must be a separate consumer POM as they can be used independently. These can be derived from the build POM. AFAIK there is no concept to explicitly provide a consumer POM for a module (and in most cases this should not be necessary as they can be derived from the build POM).

for better understanding of this plan (that would deserve more than a small discussion in an issue in Maven core: is there a doc somewhere?), is there a convention between original artifactId and automatic artifactIds?

There seems not to be much documentation about M4 consumer POM concepts or a specification how to work with Java modules (I'd be happy to find more about it).

That said, I'd be volunteering to create such concept documentation.

and is there a jar associated to the original artifactId?

For a multi-module project there is not much need to create a binary (classes) JAR for the project itself, as each module should result in its own jar. Perhaps the sources or javadocs etc. could be created from the original project. This might be subject to discussion.

[gnodet]

Hello @gnodet. I have a question related to this pull request. Is it possible to edit the consumer POM of the attached artifacts for:

• Setting the value of the <artifactId> element to the artifact.getArtifactId() value.
• Filter the dependencies for keeping only the ones that are declared in the module-info file.
• Change the <optional> value of some dependencies depending on whether the dependency is declared as static in the module-info file.

Any tip would be greatly appreciated.

The consumer POM is only generated from the main build POM for project being compiled. If we generate new artifacts dynamically, we'll have to generate the POM and attach it AFAIK. The DefaultConsumerPomBuilder uses the MavenProject as the main input, so I don't really think it makes sense to reuse it.

[desruisseaux]

Hello @hboutemy and all. First, there are a few Java 9 concepts that I would like to ensure that they are well known:

• Since Java 9, javac can compile many modules at once. This is possible when using the so-called module source hierarchy.
• There are some technical advantages in doing that, and no way I'm aware of to get some of these advantages in the traditional way.
• When using module hierarchy, the javac output is <module>/<package>/*.class. The <module> additional level is a standard Java output, not something that we decided on our side.
• The above additional <module> directory level is understood by other standard Java tools such as java and javadoc. One notable exception is the jar tool, maybe intentionally for forcing a limitation of one module per JAR for smaller download.

They are been attempts to explain these aspects on the Maven mailing list, on a wiki with proof of concept and with a real project experiment using Gradle. However, I got the feeling that the above is still nebulous to most peoples, and that the best way to advance would be to give to developers a possibility to try easily by themselves.

The Maven Compiler Plugin 4.0.0-beta-3 is already capable to compile a project organized with module source hierarchy. Consequently, the output in target/classes is multi-module when the project is organized that way (again, because javac works that way, not because we made a choice that we are imposing to others). For Java tools such as java and javadoc, this output can still be consumed as a single output with a single --module-path target/classes option. This option understand the javac module hierarchy output. Therefore, for these tools, the "1 Maven project = 1 main output" pattern could still apply. But as said above, jar is an exception (probably intentional).

Now about the questions:

what is the pom of "automatic sub-artifactId Java Module" vs the base artifactId? (…snip…) you expect to use the same consumer pom for automatic artifactIds of Java module JAR, just update the artifactId? and implicitly no build pom?

Note that when building a project with Maven 4, we already have a target/project-local-repo/${groupId}/${artifactId}/${version} directory with a ….pom and a …-consumer.pom files in it. The consumer POM seems to differ from the source POM in that it has comments removed, the <scm> and <repositories> sections removed, the ${…} properties resolved to their values and some reformatting applied. My proposal is to replicate the consumer POM for each JAR with the following changes:

• artifactId set to the Java module name.
• Dependencies filtered to the intersections of the Maven dependencies and the module-info:
  • Java modules enumerated in a requires statement are ordinary dependency (compile scope).
  • Java modules enumerated in a static requires statement are dependency with the optional element set to true.
• <build> and <profiles> sections removed.

Therefore, the base artifactId would not be used except for the build (original) POM.

is there a jar associated to the original artifactId?

No when using the module source hierarchy. Reminder: nothing change when using the traditional package hierarchy.

when I read "This is needed by the new Maven JAR Plugin proposal apache/maven-jar-plugin#508.", I don't get the relationship with using jar vs MavenArchiver: we need to be cautious

The new proposed JAR plugin automatically detects when module hierarchy is used, and produces one JAR per module in that case. However, it results in an exception thrown by Maven core if the relaxation proposed in this pull request is not applied. Reminder: again, nothing change when the JAR plugin detects that the traditional package hierarchy was used.

The reason why the JAR pull request mixes two tasks in the same pull request is because one of the reasons to migrate to the jar tool was to handle multi-release JAR files with the --release option in order to address apache/maven-jar-plugin#484 and by experience with the compiler plugin (verified again with the JAR plugin), it is easier to address multi-release and multi-module together than separately.

[ascheman]

• Since Java 9, javac can compile many modules at once. This is possible when using the so-called module source hierarchy.
• There are some technical advantages in doing that, and no way I'm aware of to get some of these advantages in the traditional way.
• When using module hierarchy, the javac output is <module>/<package>/*.class. The <module> additional level is a standard Java output, not something that we decided on our side.
• The above additional <module> directory level is understood by other standard Java tools such as java and javadoc. One notable exception is the jar tool, maybe intentionally for forcing a limitation of one module per JAR for smaller download.

They are been attempts to explain these aspects on the Maven mailing list, on a wiki with proof of concept and with a real project experiment using Gradle. However, I got the feeling that the above is still nebulous to most peoples, and that the best way to advance would be to give to developers a possibility to try easily by themselves.

Would you mind if we add a respective (sub-) page to the Confluence as documentation of this background to Java modules?

[desruisseaux]

Would you mind if we add a respective (sub-) page to the Confluence as documentation of this background to Java modules?

That would be welcome. One of the biggest difficulty we are facing with Java modules is explaining them. More documentation attempts may help.

[ascheman]

is there a jar associated to the original artifactId?

No when using the module source hierarchy. Reminder: nothing change when using the traditional package hierarchy.

when I read "This is needed by the new Maven JAR Plugin proposal apache/maven-jar-plugin#508.", I don't get the relationship with using jar vs MavenArchiver: we need to be cautious

The new proposed JAR plugin automatically detects when module hierarchy is used, and produces one JAR per module in that case. However, it results in an exception thrown by Maven core if the relaxation proposed in this pull request is not applied. Reminder: again, nothing change when the JAR plugin detects that the traditional package hierarchy was used.

To make clear that the relaxation is only necessary and used in the case of multi-module JARs, the proposed change could be adapted to check whether the project is multi-module and only leave out the comparison of artifactId in this case (as suggested before):

...
  String a1 = project.getArtifactId();
  String a2 = artifact.getArtifactId();
...
  // ArtifactId may differ only for multi-module projects (source roots with module name)
  boolean isMultiModule = getSourceRoots(project).stream().anyMatch(sr -> sr.module().isPresent());
  if (!Objects.equals(a1, a2) && !isMultiModule) {
      throw new IllegalArgumentException(...);
  }
...

[desruisseaux]

I pushed a variation of @ascheman proposal: in addition of allowing the artifactId to differ only when multi-module is used, it also check that the new artifactId is the name of one of the Java module.

[hboutemy]

the relaxation is only necessary and used in the case of multi-module JARs

I'm starting to understand that one single multi-java module build gives many "what used to be classical jars, with one build (= 1 pom) per jar"

then this is not about relaxing attach strategy
it's about creating pseudo POMs (with their distinct gav) that will host their jars done from one central build

now I get that we have one build pom, that generates many consumer poms with their own output modular jar.
Saying that, is "multi-module JARs" a good description? each jar is a single module, but it is a multi-Java module build done (with one single compiler invocation), isn't it?

[hboutemy]

one question: one reason for having one build pom per sub-project is that there are resources, assemblies, and everything that goes beyond just the compiler step, and also there are unit tests, and ITs, and everything else

in this "one single pom for one single compiler run that build many independent Java modular jars", how do you handle the flexibility about everything that is not .class in the modular jars, and their tests, etc?

[hboutemy]

@gnodet I don't get how v4 API permits to attach something to a pom (in v3 a MavenProject) that does not exist
who will create the pom that will be necessary for install/deploy?

[gnodet]

@gnodet I don't get how v4 API permits to attach something to a pom (in v3 a MavenProject) that does not exist who will create the pom that will be necessary for install/deploy?

This is not limited to the v4 API. The jar plugin will attach the artifacts and the poms when building a modular project. AFAIK central will reject any upload for an artifact without any matching pom. We could enforce that at install/deploy time in resolver if we want.

[gnodet]

one question: one reason for having one build pom per sub-project is that there are resources, assemblies, and everything that goes beyond just the compiler step, and also there are unit tests, and ITs, and everything else

in this "one single pom for one single compiler run that build many independent Java modular jars", how do you handle the flexibility about everything that is not .class in the modular jars, and their tests, etc?

The <source> element in POM 4.1.0 is more flexible than the <resource> element in POM 4.0.0. This is where all the modular information is stored.

[hboutemy]

The jar plugin will attach the artifacts and the poms when building a modular project

uh, attaching the pom?
I see how a plugin can attach a consumer pom to a Maven Project that provides the build pom: but we still miss the Maven Project that provides the build pom

The element in POM 4.1.0 is more flexible than the element in POM 4.0.0. This is where all the modular information is stored.

I'm still not convinced <source> can replace what we configure with plugins in a sub-project

[gnodet]

The jar plugin will attach the artifacts and the poms when building a modular project

uh, attaching the pom? I see how a plugin can attach a consumer pom to a Maven Project that provides the build pom: but we still miss the Maven Project that provides the build pom

I don't understand. You can attach any kind of artifact to a project, it can be a jar, a pom, or whatever. From the install/deployment POV, there's no difference between a main pom, a consumer pom, or the jar produced. Those are all artifacts (groupId, artifactId, version, extension, classifier) that point to file on the filesystem. Then, if we talk about semantics, I'm not sure yet how to produce those poms, but I think the overall idea is to create something similar to the consumer POM for the project, with small modifications to follow the rules of a specific JPMS module (not a maven module).

The element in POM 4.1.0 is more flexible than the element in POM 4.0.0. This is where all the modular information is stored.

I'm still not convinced <source> can replace what we configure with plugins in a sub-project

I don't think that's a goal. We're not talking about subprojects here, just JPMS modules that are created from a single POM file. Maybe you're thinking about plugins that would generate other kind of resources ? For example a plugin that would read the class files and generate something to be included in the jar ? That would definitely not work without adapting the plugin to support this JPMS module layout. Or you could configure multiple executions, each one pointing to the specific output directory of the module. For example with the modular project IT, we'll have target/classes/org.bar and target/classes/org.foo, so either the plugin supports this new layout and it will figure it out, or you can create two executions explicitly pointing to those directories.

[desruisseaux]

Saying that, is "multi-module JARs" a good description? each jar is a single module, but it is a multi-Java module build done (with one single compiler invocation), isn't it?

Correct, each JAR is a single module, and the multi-module build is produced by one single invocation of the compiler. I agree that "multi-module JARs" may be confusing. "Module hierarchy" when talking about the directory layout, or just "multi-module" otherwise, would probably be safer.

one question: one reason for having one build pom per sub-project is that there are resources, assemblies, and everything that goes beyond just the compiler step, and also there are unit tests, and ITs, and everything else

Resources are supported by the new <source> element. For JUnit tests, a command like java --module-path target/classes accepts the module hierarchy as a whole, so it will still like a single output from Maven Surefire point of view. For assembly, if we are talking about things like "fat jar", they are not compatible with Java modules as far as I know because I don't see how we could merge all the module-info in a single one without altering the semantic.

I'm still not convinced can replace what we configure with plugins in a sub-project

Indeed, <source> does not replace everything. It replaces <sourceDirectory>, <testSourceDirectory>, <resources>, <testResources> and <scriptSourceDirectory>. Other plugin configurations stay in the plugin (except the ones for specifying some kinds of dependencies such as processors, but this is another topic not related to Java modules).

[desruisseaux]

Correction to the above: <source> replaces also <release> in the plugin configuration, but this is a special case.

[hboutemy]

thank you @gnodet and @desruisseaux for your explanations:

• modular-sources compiler IT also helped me a lot
• I still need to dig into https://github.com/Geomatys/maven/wiki , but it's very valuable, for example https://github.com/Geomatys/maven-compiler-plugin/wiki/User-guide

I now see the wider picture: at compile, test and jar level, I see that this modular-sources approach improvements have been done or are planned (I suppose there is an equivalent IT in Surefire and in Maven Jar Plugin? I don't see in https://github.com/apache/maven-jar-plugin/tree/master/src/it , I ignore fore now Surefire as its code is too complex for me :) )

I'm still not yet completely convinced at install/deploy level: I'll try to dig into my feeling to be more actionable whatever I discover when going more in-depth

no modular sources IT in https://github.com/apache/maven-install-plugin/tree/master/src/it nor https://github.com/apache/maven-deploy-plugin/tree/master/src/it : this is one of my biggest concerns for now

I'm working on many plugins that iterate over everything that will be installed/deployed: Maven Artifact Plugin, CycloneDX and SPDX Maven Plugins
Typical algorithm in a simple case is https://github.com/apache/maven-artifact-plugin/blob/maven-artifact-plugin-3.6.1/src/main/java/org/apache/maven/plugins/artifact/buildinfo/DescribeBuildOutputMojo.java#L123
= iterate over MavenProject p : session.getProjects(), then iterate over:

• p.getFile() for the pom file
• p.getArtifact().getFile() for the main artifact file
• p.getAttachedArtifacts() for attached "sub-" artifacts
• and there is the trick in Maven 4 between the build pom (p.getFile()) and it associated consumer pom (one attached sub-artifact)

Currently, GAV coordinates from Artifacts are the same as MavenProject from where they come

With one Maven project producing many modular jars based on their modular source structure, now I get that there will be artifactIds deducted from module name in modular sources
I hope we'll check that there is no conflict between MavenProjects and attachements to a modular build, as an attachment could be done at both levels

I won't stay more theoretical, it become too long;
I see here that modular sources support requires to change the whole plugins ecosystem and also Maven core, and for now we see each separate change, but not a modular sources project on all his build phases = what I'm trying to imagine and check, particularly at install and deploy levels = where it starts to hit Maven repository format and Maven Central (that is not a must for Gradle, but is for Maven)

Is there a common sample modular sources project that is used to test all these changes in a common team approach?
Then we can link to all the updates that are needed to the many plugins involved in the build lifecycle

And last question: from Maven core perspective, does it target Maven 4.1 only, or is it expected to be backported to some previous branches?

one partial conclusion in addition is that we'll need to keep https://github.com/Geomatys/maven/wiki at Maven level, at least partially, because this is really the best design doc I now see
(and previously, I did not have/take time, then I overlooked many topics, sorry @desruisseaux if my questions are sometimes redundant with what you already wrote: it's such a whole new world to get in...)

[hboutemy]

issue title should be more restrictive than just "relax" but at least something talking about modular sources (good news for me: we don't relax and let anything happen, but replace with another well defined constraint)

"support modular sources: " prefix?
"accept attached artifactId from module different from project's one"

[gnodet]

thank you @gnodet and @desruisseaux for your explanations:

• modular-sources compiler IT also helped me a lot
• I still need to dig into https://github.com/Geomatys/maven/wiki , but it's very valuable, for example https://github.com/Geomatys/maven-compiler-plugin/wiki/User-guide

I now see the wider picture: at compile, test and jar level, I see that this modular-sources approach improvements have been done or are planned (I suppose there is an equivalent IT in Surefire and in Maven Jar Plugin? I don't see in https://github.com/apache/maven-jar-plugin/tree/master/src/it , I ignore fore now Surefire as its code is too complex for me :) )

I'm not sure about which changes will be required in surefire yet. I don't know that code base well either.

I'm still not yet completely convinced at install/deploy level: I'll try to dig into my feeling to be more actionable whatever I discover when going more in-depth

no modular sources IT in https://github.com/apache/maven-install-plugin/tree/master/src/it nor https://github.com/apache/maven-deploy-plugin/tree/master/src/it : this is one of my biggest concerns for now

This should not require any change at all. Install / deploy just take attached artifacts and delegate to the resolver for installing / deploying. This very PR is exactly about being able to attach those generated artifacts to the project, without being limited to the same artifactId. This is the only required change AFAIK.

I'm working on many plugins that iterate over everything that will be installed/deployed: Maven Artifact Plugin, CycloneDX and SPDX Maven Plugins Typical algorithm in a simple case is https://github.com/apache/maven-artifact-plugin/blob/maven-artifact-plugin-3.6.1/src/main/java/org/apache/maven/plugins/artifact/buildinfo/DescribeBuildOutputMojo.java#L123 = iterate over MavenProject p : session.getProjects(), then iterate over:

• p.getFile() for the pom file
• p.getArtifact().getFile() for the main artifact file
• p.getAttachedArtifacts() for attached "sub-" artifacts
• and there is the trick in Maven 4 between the build pom (p.getFile()) and it associated consumer pom (one attached sub-artifact)

Currently, GAV coordinates from Artifacts are the same as MavenProject from where they come

Yes, and all the JAR/POMs for the modules will be available from p.getAttachedArtifacts() which returns a list of Artifact which will reflect everything. The only problem is if a plugin assumes that the artifactId is the same as the project, or something like that.

With one Maven project producing many modular jars based on their modular source structure, now I get that there will be artifactIds deducted from module name in modular sources I hope we'll check that there is no conflict between MavenProjects and attachements to a modular build, as an attachment could be done at both levels

Good point. This has to be checked. This would mean we have conflicting module names somehow I suppose. @desruisseaux ?

I won't stay more theoretical, it become too long; I see here that modular sources support requires to change the whole plugins ecosystem and also Maven core, and for now we see each separate change, but not a modular sources project on all his build phases = what I'm trying to imagine and check, particularly at install and deploy levels = where it starts to hit Maven repository format and Maven Central (that is not a must for Gradle, but is for Maven)

I really don't expect any change for installation/deployment. The only changes is for plugins that would need to adapt to the JPMS layout.

Is there a common sample modular sources project that is used to test all these changes in a common team approach? Then we can link to all the updates that are needed to the many plugins involved in the build lifecycle

And last question: from Maven core perspective, does it target Maven 4.1 only, or is it expected to be backported to some previous branches?

Maven-core has provided the needed infrastructure with the v4 API in 4.0.x, but most of the work is in the v4 core plugins (resources, compiler, jar...). A project migrated from Maven 3 won't see any change.

one partial conclusion in addition is that we'll need to keep https://github.com/Geomatys/maven/wiki at Maven level, at least partially, because this is really the best design doc I now see (and previously, I did not have/take time, then I overlooked many topics, sorry @desruisseaux if my questions are sometimes redundant with what you already wrote: it's such a whole new world to get in...)

[ascheman]

Is there a common sample modular sources project that is used to test all these changes in a common team approach?
Then we can link to all the updates that are needed to the many plugins involved in the build lifecycle

Well, we have a collection of about 40 samples of Java Modules use cases. 35 of them (all applicable ones) have been migrated to Maven 4 (or its current state of support for Java Modules). And we already found out some (new) challenges and verified what works by now, e.g.., compiler support is very good by now.

One of the open challenges is the incomplete support of resources in the new sources tree of the POM -> PR #11505.

[ascheman]

In general, let me remind you of some ongoing documentation of Java Modules support in Confluence.

[desruisseaux]

I don't see in https://github.com/apache/maven-jar-plugin/tree/master/src/it

An integration test is provided with the apache/maven-jar-plugin#508 pull request, but it depends on this PR. For other plugins, the current state is in the link provided by Gerd in above comment.

Is there a common sample modular sources project that is used to test all these changes in a common team approach?

In addition to the samples given by Gerd in above comment, I use a clone of the Apache SIS project where the Gradle build has been replaced by Maven 4. The features tested are:

• Compilation of main code.
• Compilation of test code with module-info-patch.maven.
• Creation of JAR files (using the above-cited pull request).
• Some modules targeting different Java version.

The result is much better than what we could achieve by hacking the Gradle build. In particular, we had to create some org.apache.sis.test modules only for adding module-info managing the dependencies in a way that we couldn't achieve with Gradle configuration. This hack is gone with Maven 4, the build has never been so clean.

from Maven core perspective, does it target Maven 4.1 only, or is it expected to be backported to some previous branches?

The work is backported to 4.0.x, but as Guillaume said, the changes that are done in Maven core are small adjustments. The biggest changes where done about one year ago, with the introduction of <source> element and the use of <type> for controlling the path where to put dependencies. Now, the largest works are in plugins.

we'll need to keep https://github.com/Geomatys/maven/wiki at Maven level

Yes, this is also planed. I already started an updated version of the user guide, but I'm not sure where to put it as it is of wider scope than only the compiler plugin.

With one Maven project producing many modular jars based on their modular source structure, now I get that there will be artifactIds deducted from module name in modular sources I hope we'll check that there is no conflict between MavenProjects and attachements to a modular build, as an attachment could be done at both levels

Good point. This has to be checked. This would mean we have conflicting module names somehow I suppose.

Indeed, a check would be desirable, maybe in a separated pull request. For now, the risk is slightly reduced by naming tradition (not a formal convention), as Maven artifacts often have a - in their name, which is illegal in a Java module name.