Untarring maven assembly archive into bundle module

Question

Hi

I have several sub module (say A, B + C) that each generate an assembly tar.gz and a module (X) that will bundle them all together by extracting the tarballs into the target directory.

Right now setting up my dependency on A, B + C using the classifier and type in module X POM and using an assembly file with dependencySets the module does not seem to be pulling the tarball from my local repo, rather rebuilding it.

This is causing issues because A, B + C each has it's own filters. When X rebuilds A it leaves it unfiltered. I want maven to just either get the A-distro.tar.gz from my repo or if not present rebuild A, put it in the repo then have X use this.

Here is the assembly.xml file I'm using. I couldn't get moduleSets to work when using the coordinates so went for this:

<assembly>
    <id>distro</id>
    <formats>
        <format>dir</format>
        <format>tar.gz</format>
    </formats>
    <baseDirectory>${project.version}</baseDirectory>
    <includeBaseDirectory>false</includeBaseDirectory>

    <dependencySets>

        <dependencySet>
            <useTransitiveFiltering>true</useTransitiveFiltering>
            <useProjectArtifact>false</useProjectArtifact>
            <unpack>false</unpack>
            <outputDirectory>/lib</outputDirectory>
            <excludes>
                <exclude>*:tar.gz</exclude>
            </excludes>
        </dependencySet>

        <dependencySet>
            <useTransitiveFiltering>true</useTransitiveFiltering>
            <useProjectArtifact>false</useProjectArtifact>
            <unpack>true</unpack>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>*:tar.gz</include>
            </includes>
        </dependencySet>

    </dependencySets>

</assembly>

And POM for project X:

<project>
  <parent>
    <artifactId>parent_project</artifactId>
    <groupId>org.myorg</groupId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.myorg</groupId>
  <artifactId>X</artifactId>
  <packaging>pom</packaging>
  <name>X</name>
  <version>1.0-SNAPSHOT</version>
  <build>
          <plugins>
                <plugin>
                        <artifactId>maven-assembly-plugin</artifactId>
                </plugin>
          </plugins>

    </build>

  <dependencies>
          <dependency>
              <groupId>org.myorg</groupId>
              <artifactId>A</artifactId>
              <classifier>distro</classifier>
              <type>tar.gz</type>
        </dependency>
  </dependencies>
</project>

---

(RS moved content from OP's answer)

ok thanks for replying.

I've created a test project using the principle you describe of having one module for assembly, one for aggregator:

./pom.xml
./bundle
./bundle/pom.xml
./bundle/src
./bundle/src/main
./bundle/src/main/assembly
./bundle/src/main/assembly/assembly-files.xml
./bundle/src/main/assembly/assembly.xml
./module1
./module1/pom.xml
./module1/src
./module1/src/main
./module1/src/main/assembly
./module1/src/main/assembly/assembly-files.xml
./module1/src/main/assembly/assembly.xml
./module1/src/main/conf
./module1/src/main/java
./module1/src/main/java/org
./module1/src/main/java/org/test
./module1/src/main/java/org/test/Test.java
./module2
./module2/pom.xml
./module2/src
./module2/src/main
./module2/src/main/java
./module2/src/main/java/org
./module2/src/main/java/org/test
./module2/src/main/java/org/test/Test.java

Module 1 produces an assembly file of the following coordinates:

        <dependency>
            <groupId>org.test</groupId>
            <artifactId>module1</artifactId>
            <version>1.0-SNAPSHOT</version>
            <classifier>distro</classifier>
            <type>tar.gz</type>
        </dependency>

The following file is produced:

~/mvnrepos/org/test/module1/1.0-SNAPSHOT/module1-1.0-SNAPSHOT-distro.tar.gz

When building the bundle with the following assembly the tar.gz is pulled in and used:

<assembly>
    <id>distro</id>
    <formats>
        <format>dir</format>
        <format>tar.gz</format>
    </formats>
    <baseDirectory>${project.version}</baseDirectory>
    <includeBaseDirectory>false</includeBaseDirectory>


    <dependencySets>

        <dependencySet>
            <includes>
                <include>org.test:module1:tar.gz</include>
            </includes>
            <useTransitiveFiltering>true</useTransitiveFiltering>
            <useProjectArtifact>false</useProjectArtifact>
            <unpack>true</unpack>
            <unpackOptions>
                <excludes>
                        <exclude>lib/</exclude>
                </excludes>
            </unpackOptions>
            <outputDirectory>lib/</outputDirectory>
        </dependencySet>

    </dependencySets>

<!-- just includes conf and bin dirs -->
    <componentDescriptors>
        <componentDescriptor>src/main/assembly/assembly-files.xml</componentDescriptor>
    </componentDescriptors>

</assembly>

However if I cleardown my repository and clean the project at the root so the tar.gz is removed completely, when I change into the bundle dir and mvn install it fails as maven cannot work out it needs to rebuild module 1 in the absence of the tar.gz in order to get it. Here is my bundle pom:

<project>
  <parent>
    <artifactId>test_parent</artifactId>
    <groupId>org.test</groupId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.test</groupId>
  <artifactId>bundle</artifactId>
  <packaging>pom</packaging>
  <name>bundle</name>
  <version>1.0-SNAPSHOT</version>
  <url>http://maven.apache.org&lt;/url&gt;
  <build>
          <plugins>
                <plugin>
                        <artifactId>maven-assembly-plugin</artifactId>
                </plugin>
          </plugins>
    </build>

  <dependencies>

            <dependency>
                <groupId>org.test</groupId>
                <artifactId>module1</artifactId>
            </dependency>

            <dependency>
                <groupId>org.test</groupId>
                <artifactId>module2</artifactId>
            </dependency>

            <dependency>
                <groupId>org.test</groupId>
                <artifactId>module1</artifactId>
                <version>1.0-SNAPSHOT</version>
                <classifier>distro</classifier>
                <type>tar.gz</type>
                <scope>runtime</scope>
            </dependency>

  </dependencies>
</project>

How can I get maven to realize that the assembly must be rebuilt by rebuilding module 1?

I'm trying to keep each assembly within a separate module so each module can be built and tested as a separate unit rather than having an aggregator which needs to be rebuilt in order to run any of the modules. Then if the a full run is required the bundle can be built by simply untarring each assembly tarball into a directory under bundle/target.

Perhaps the above is going against the grain - but this seems to be the thrust of the maven assembly chapter in the Sonotype book:

http://www.sonatype.com/books/maven-book/reference/assemblies-sect-assembling-via-depend.html

Unfortunately the examples zip provided by sonotype has nothing under chapter 12 :-(

Of course I could just break out into ant and use this but would like to implement purely in maven if at all possible.

btw maven-assembly-plugin version is 2.2-beta-2.

Answer 1

I'm not sure I understand your problem so let me try to rephrase it.

You have 4 projects, A, B, C and X. You have A, B, and C defined as sub-projects of X When you build project X you want to include the tar.gz files for projects A, B and C.

The problem is that you want to build A only if required, and download it from the project repository if it is already present?

Assuming I understand the problem correctly, I'm still not clear how you expect the children to be conditionally built. If X declares the others as modules Maven will always build them along with the parent. If X declares the others as dependencies they will only ever be treated as dependencies, so the build will fail unless the artifacts exist in the repository (i.e. you've already built them).

---

If you want to be able to exercise flexibility over the build, I'd recommend separating the responsibilities of assembly and aggregator into different projects.

The aggregator simply declares the projects as modules so you can conveniently build all the projects in one command.

The assembly project (your project X) declares dependencies on projects A, B and C. When you build project X it is required that the others have already been built, but if built in the aggregator, the reactor will have built them first. Whereas if you build it standalone, you know the others are up to date.

---

Some additional points on Maven usage that should clarify this approach.

The Maven repository is a record of all the artifacts you've developed. You don't need to clear the local repository between builds. It is intended to act as a repository of all the artifacts. So if you install project A, that is the version that will always be used (unless and until you rebuild project A).

If you do have to clear the local repository, you should deploy your artifacts to a remote repository (see the relevant section of the Nexus book for guidance on getting started) so they can be incorporated into your build for project X even if the local repository is cleared.

So the workflow for building the projects is:

1. Run mvn install on the aggregator project or the relevant project (A,B, or C).
2. Projects are installed to the local repository.
3. Don't clear the local repository!
4. Run mvn install on project X, Maven will retrieve the dependencies from the repository and package them into your assembly.

If you use mvn deploy, the same principles apply, but because the artifacts are in the remote repository you have freedom to clear the local repository.

To emphasise my earlier points:

• There is no mechanism to conditionally build modules. They either build or they don't (however some steps of the build might be skipped if they are not needed, e.g. compilation might have already been done in a previous build)
• There is no need to purge the local repository (and if you do, you should have the modules deployed to a remote repository so they can be retrieved).

Answer 2

moved answer into question, rollback if you're not happy with this

Answer 3

Hi

ok I should have been more clear on why I was clearing down the local repo - I understand this is not a normal step as part of the build cycle.

I was clearing down my local repo to emulate what would happen if I tried a build for the very first time and nothing exists on a remote repo. This is because on checking out the parent + modules all the code I need is present.

Parent
\----module1 (includes assembly classifier)
|
\----module2
|
\----bundle

eg build with empty repo from parentbuilds module1, module2 then bundle. All works well in this order:

parent, module1, module2, bundle

however if I have an empty repos and cd to X then build maven cannot work out that it needs to build the parent which depends on module1 and module2, so if possible I'd like it to build in this order:

parent, module1, module2, bundle

I think this is not possible in maven as you say there is no mechanism for conditional module builds! I thought maven would support this as it has info on the parent and the parent has info on the children, but I think I was inferring too much.

Many thanks for your replies Rich. Getting there with maven but it is like pulling teeth, in large part due to the style of the sonotype manual.

ps editing of questions looses context. I didn't realise stackoverflow did this. usenet style had it right first time, write your own reply, include references to former replies if required, post at the bottom. Editing prior posts IMHO erases the flow of a conversation - for some issues the journey to the solution is as instructive as the destination! this last paragraph is 400 chars :-)

eg I could not put this comment in as recommended because it goes over the technical sound barrier of 600 characters.