Learning Makefiles as a Java developer. Part 6: resolving transitive dependencies

Marcio Endo

January 14, 2024

In the previous post of this series we introduced the concept of "resolution files". It is part of our designed solution for handling transitive dependencies in our Makefile.

Our solution implementation is still not complete. The resolution files work but we had to create them manually.

In this sixth and final part of our series we will improve the implementation of our dependency handling solution.

Our goal is to automatically create the resolution files. In order to do it, we will also have to automatically:

resolve the transitive dependencies;
download the JAR files for the resolved transitive dependencies; and
download the JAR files for the declared direct dependencies.

Let's continue.

Iteration #17: an application to resolve transitive dependencies

Let's summarize our current requirements.

We have the GAV coordinate of a particular dependency. For example, we have the GAV for the jackson-databind Java library:

com.fasterxml.jackson.core/jackson-databind/2.16.1

We have to resolve the transitive dependencies. In this case they are:

com.fasterxml.jackson.core/jackson-annotations/2.16.1
com.fasterxml.jackson.core/jackson-core/2.16.1

Next, we have to download all of the JAR files into our local repository. And the paths of all of the downloaded artifacts:

repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar

Needs to be persisted in the corresponding "resolution file" located at:

resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1

To accomplish this task we will use the Resolver.java file of the objectos.mk project.

The `Resolver.java` file

The Resolver.java file encapsulates the Apache Maven Artifact Resolver in a single Java file. It is designed to be run using JEP 330 like so:

java --class-path $(RESOLVER_CLASS_PATH) Resolver.java \
    --local-repo $(LOCAL_REPO_PATH) \
    --resolution-dir $(DEPS) \
    com.fasterxml.jackson.core/jackson-databind/2.16.1

We will not discuss its implementation in this blog post.

You can find its source code here.

Our resolver class path

We will use the Resolver.java file to resolve our dependencies. But to run the file we need to first download its dependencies.

We are in a sort of a chicken and egg situation.

For this reason, the Resolver.java file lives in a Maven project. Which means we can run the following:

mvn --file objectos.mk/resolver/pom.xml \
	-Dscope=runtime \
	-DoutputFile=/tmp/resolve.txt \
	-Dtokens=whitespace \
	dependency:tree

It produces the output in the /tmp/resolve.txt file.

We can translate the output to the following Makefile variable:

## Resolver.java deps
RESOLVER_DEPS  = commons-codec/commons-codec/1.16.0
RESOLVER_DEPS += org.apache.commons/commons-lang3/3.12.0
RESOLVER_DEPS += org.apache.httpcomponents/httpclient/4.5.14
RESOLVER_DEPS += org.apache.httpcomponents/httpcore/4.4.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-api/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-connector-basic/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-impl/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-named-locks/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-spi/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-supplier/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-transport-file/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-transport-http/1.9.16
RESOLVER_DEPS += org.apache.maven.resolver/maven-resolver-util/1.9.16
RESOLVER_DEPS += org.apache.maven/maven-artifact/3.9.4
RESOLVER_DEPS += org.apache.maven/maven-builder-support/3.9.4
RESOLVER_DEPS += org.apache.maven/maven-model-builder/3.9.4
RESOLVER_DEPS += org.apache.maven/maven-model/3.9.4
RESOLVER_DEPS += org.apache.maven/maven-repository-metadata/3.9.4
RESOLVER_DEPS += org.apache.maven/maven-resolver-provider/3.9.4
RESOLVER_DEPS += org.codehaus.plexus/plexus-interpolation/1.26
RESOLVER_DEPS += org.codehaus.plexus/plexus-utils/3.5.1
RESOLVER_DEPS += org.slf4j/jcl-over-slf4j/1.7.36
RESOLVER_DEPS += org.slf4j/slf4j-api/1.7.36
RESOLVER_DEPS += org.slf4j/slf4j-nop/1.7.36

To derive a class path value we use the functions we created in part 4:

## Resolver local repo jars
RESOLVER_DEPS_JARS = $(call gavs-to-local,$(RESOLVER_DEPS))

## Resolver class-path
RESOLVER_CLASS_PATH = $(call class-path,$(RESOLVER_LOCAL_JARS))

Next, we need to update our Makefile so it downloads all these files.

A rule to automatically download the files

We add the following rule and associated variables to our Makefile:

## Where to find our Resolver.java source 
RESOLVER_URL = https://raw.githubusercontent.com/objectos/objectos.mk/3ff20b750265bb1bbfd08c86e2418e7300d232d8/resolver/src/main/java/Resolver.java

## Resolver.java path
RESOLVER_JAVA = Resolver.java

$(RESOLVER_JAVA): $(RESOLVER_DEPS_JARS)
	wget --no-verbose $(RESOLVER_URL)

The prerequisites of the rule are the JAR files needed to run our resolver. Remember that in part 4 we declared a rule to automatically download JAR files from the remote repository.

The recipe of the rule downloads the Resolver.java from GitHub using the wget tool.

Let's test it.

First, let's check the contents of our working directory:

$ find -type f
./Makefile
./repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
./repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
./repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
./repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
./main/objectos/library/Hello.java
./main/objectos/library/Hi.java
./main/objectos/library/Say.java
./resolution/org.slf4j/slf4j-api/1.7.36
./resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1

Next, let's run the Resolver.java target:

make Resolver.java

And here's part of the console output:

wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
2024-01-13 11:46:20 URL:https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar [587402/587402] -> "repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar" [1]
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar
2024-01-13 11:46:20 URL:https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar [785639/785639] -> "repository/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar" [1]
(...)
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar
2024-01-13 11:46:22 URL:https://repo.maven.apache.org/maven2/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar [3946/3946] -> "repository/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar" [1]
wget --no-verbose https://raw.githubusercontent.com/objectos/objectos.mk/3ff20b750265bb1bbfd08c86e2418e7300d232d8/resolver/src/main/java/Resolver.java 
2024-01-13 11:46:22 URL:https://raw.githubusercontent.com/objectos/objectos.mk/3ff20b750265bb1bbfd08c86e2418e7300d232d8/resolver/src/main/java/Resolver.java [7250/7250] -> "Resolver.java" [1]

Let's check the contents of our working directory once again:

$ find -type f
./Makefile
./repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
./repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
./repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
./repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
./repository/org/slf4j/jcl-over-slf4j/1.7.36/jcl-over-slf4j-1.7.36.jar
./repository/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar
./repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
./repository/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar
./repository/org/apache/httpcomponents/httpcore/4.4.16/httpcore-4.4.16.jar
./repository/org/apache/maven/resolver/maven-resolver-api/1.9.16/maven-resolver-api-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-connector-basic/1.9.16/maven-resolver-connector-basic-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-impl/1.9.16/maven-resolver-impl-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-named-locks/1.9.16/maven-resolver-named-locks-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-spi/1.9.16/maven-resolver-spi-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-supplier/1.9.16/maven-resolver-supplier-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-transport-file/1.9.16/maven-resolver-transport-file-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-transport-http/1.9.16/maven-resolver-transport-http-1.9.16.jar
./repository/org/apache/maven/resolver/maven-resolver-util/1.9.16/maven-resolver-util-1.9.16.jar
./repository/org/apache/maven/maven-artifact/3.9.4/maven-artifact-3.9.4.jar
./repository/org/apache/maven/maven-builder-support/3.9.4/maven-builder-support-3.9.4.jar
./repository/org/apache/maven/maven-model-builder/3.9.4/maven-model-builder-3.9.4.jar
./repository/org/apache/maven/maven-model/3.9.4/maven-model-3.9.4.jar
./repository/org/apache/maven/maven-repository-metadata/3.9.4/maven-repository-metadata-3.9.4.jar
./repository/org/apache/maven/maven-resolver-provider/3.9.4/maven-resolver-provider-3.9.4.jar
./repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar
./repository/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.jar
./repository/commons-codec/commons-codec/1.16.0/commons-codec-1.16.0.jar
./main/objectos/library/Hello.java
./main/objectos/library/Hi.java
./main/objectos/library/Say.java
./resolution/org.slf4j/slf4j-api/1.7.36
./resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1
./Resolver.java

It works. It downloaded:

the dependencies of our resolver; and
the Resolver.java file itself.

Iteration 18: integrating `Resolver.java` into our build

Let's use the Resolver.java file to generate our "resolution files".

In order to do it we will update our Makefile.

Generating resolution files automatically

Here's a rule for creating our resolution files:

## resolve java command
RESOLVEX  = java
RESOLVEX += --class-path $(RESOLVER_CLASS_PATH)
RESOLVEX += $(RESOLVER_JAVA)
RESOLVEX += --local-repo $(LOCAL_REPO_PATH)
RESOLVEX += --resolution-dir $(DEPS)

$(DEPS)/%: $(RESOLVER_JAVA)
	$(RESOLVEX) $(@:$(DEPS)/%=%)

The prerequisite of the rule is our Resolver.java file.

So, before resolving the first dependency of our project, Make will run the recipe for Resolver.java. As discussed in the previous section, it will download:

all of the Resolver.java dependencies; and
the Resolver.java file itself.

Then it will run the Resolver.java program for each declared dependency.

So, for each declared direct dependency, the program will:

resolve the transitive dependencies;
download the JAR files for all of the resolved transitive dependencies;
download the JAR file of the direct dependency itself; and
generate the corresponding resolution file.

Cleaning our working directory

To make sure that our updated Makefile performs all of the required tasks let's clean our project:

$ make clean
rm -rf work

We should also delete our local repository. We will also delete the resolution files we created manually in the previous post:

$ rm -r resolution/ repository/

So our working directory contains only our Makefile and the Java source files:

$ find -type f
./main/objectos/library/Hello.java
./main/objectos/library/Hi.java
./main/objectos/library/Say.java
./Makefile

Next, let's test our updated Makefile.

Testing our updated Makefile

Let's test our updated Makefile by invoking Make:

$ make

And here's a commented view of the console output.

First, it downloads the dependencies for the Resolver.java:

wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.16.0/commons-codec-1.16.0.jar
2024-01-13 17:13:38 URL:https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.16.0/commons-codec-1.16.0.jar [360738/360738] -> "repository/commons-codec/commons-codec/1.16.0/commons-codec-1.16.0.jar" [1]
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
2024-01-13 17:13:38 URL:https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar [587402/587402] -> "repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar" [1]
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar
(...)
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
2024-01-13 17:13:40 URL:https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar [41125/41125] -> "repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar" [1]
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar
2024-01-13 17:13:40 URL:https://repo.maven.apache.org/maven2/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar [3946/3946] -> "repository/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar" [1]

Next, it downloads the Resolver.java file itself:

wget --no-verbose https://raw.githubusercontent.com/objectos/objectos.mk/3ff20b750265bb1bbfd08c86e2418e7300d232d8/resolver/src/main/java/Resolver.java 
2024-01-13 17:13:40 URL:https://raw.githubusercontent.com/objectos/objectos.mk/3ff20b750265bb1bbfd08c86e2418e7300d232d8/resolver/src/main/java/Resolver.java [7344/7344] -> "Resolver.java" [1]

It then runs the Resolver.java against the org.slf4j/slf4j-api/1.7.36 dependency:

java --class-path repository/commons-codec/commons-codec/1.16.0/commons-codec-1.16.0.jar:repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:repository/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar:repository/org/apache/httpcomponents/httpcore/4.4.16/httpcore-4.4.16.jar:repository/org/apache/maven/resolver/maven-resolver-api/1.9.16/maven-resolver-api-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-connector-basic/1.9.16/maven-resolver-connector-basic-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-impl/1.9.16/maven-resolver-impl-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-named-locks/1.9.16/maven-resolver-named-locks-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-spi/1.9.16/maven-resolver-spi-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-supplier/1.9.16/maven-resolver-supplier-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-transport-file/1.9.16/maven-resolver-transport-file-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-transport-http/1.9.16/maven-resolver-transport-http-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-util/1.9.16/maven-resolver-util-1.9.16.jar:repository/org/apache/maven/maven-artifact/3.9.4/maven-artifact-3.9.4.jar:repository/org/apache/maven/maven-builder-support/3.9.4/maven-builder-support-3.9.4.jar:repository/org/apache/maven/maven-model-builder/3.9.4/maven-model-builder-3.9.4.jar:repository/org/apache/maven/maven-model/3.9.4/maven-model-3.9.4.jar:repository/org/apache/maven/maven-repository-metadata/3.9.4/maven-repository-metadata-3.9.4.jar:repository/org/apache/maven/maven-resolver-provider/3.9.4/maven-resolver-provider-3.9.4.jar:repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar:repository/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.jar:repository/org/slf4j/jcl-over-slf4j/1.7.36/jcl-over-slf4j-1.7.36.jar:repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar Resolver.java --local-repo repository --resolution-dir resolution org.slf4j/slf4j-api/1.7.36
Downloading org.slf4j:slf4j-api:pom:1.7.36
Downloading org.slf4j:slf4j-parent:pom:1.7.36

Next, it runs the Resolver.java against the com.fasterxml.jackson.core/jackson-databind/2.16.1 dependency:

java --class-path repository/commons-codec/commons-codec/1.16.0/commons-codec-1.16.0.jar:repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:repository/org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar:repository/org/apache/httpcomponents/httpcore/4.4.16/httpcore-4.4.16.jar:repository/org/apache/maven/resolver/maven-resolver-api/1.9.16/maven-resolver-api-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-connector-basic/1.9.16/maven-resolver-connector-basic-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-impl/1.9.16/maven-resolver-impl-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-named-locks/1.9.16/maven-resolver-named-locks-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-spi/1.9.16/maven-resolver-spi-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-supplier/1.9.16/maven-resolver-supplier-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-transport-file/1.9.16/maven-resolver-transport-file-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-transport-http/1.9.16/maven-resolver-transport-http-1.9.16.jar:repository/org/apache/maven/resolver/maven-resolver-util/1.9.16/maven-resolver-util-1.9.16.jar:repository/org/apache/maven/maven-artifact/3.9.4/maven-artifact-3.9.4.jar:repository/org/apache/maven/maven-builder-support/3.9.4/maven-builder-support-3.9.4.jar:repository/org/apache/maven/maven-model-builder/3.9.4/maven-model-builder-3.9.4.jar:repository/org/apache/maven/maven-model/3.9.4/maven-model-3.9.4.jar:repository/org/apache/maven/maven-repository-metadata/3.9.4/maven-repository-metadata-3.9.4.jar:repository/org/apache/maven/maven-resolver-provider/3.9.4/maven-resolver-provider-3.9.4.jar:repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar:repository/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.jar:repository/org/slf4j/jcl-over-slf4j/1.7.36/jcl-over-slf4j-1.7.36.jar:repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar Resolver.java --local-repo repository --resolution-dir resolution com.fasterxml.jackson.core/jackson-databind/2.16.1
Downloading com.fasterxml.jackson.core:jackson-databind:pom:2.16.1
Downloading com.fasterxml.jackson:jackson-base:pom:2.16.1
Downloading com.fasterxml.jackson:jackson-bom:pom:2.16.1
Downloading com.fasterxml.jackson:jackson-parent:pom:2.16
Downloading com.fasterxml:oss-parent:pom:56
Downloading org.junit:junit-bom:pom:5.9.2
Downloading com.fasterxml.jackson.core:jackson-annotations:pom:2.16.1
Downloading com.fasterxml.jackson.core:jackson-core:pom:2.16.1
Downloading org.junit:junit-bom:pom:5.9.3
Downloading com.fasterxml.jackson.core:jackson-databind:jar:2.16.1
Downloading com.fasterxml.jackson.core:jackson-annotations:jar:2.16.1
Downloading com.fasterxml.jackson.core:jackson-core:jar:2.16.1

From the two generated resolution files, it generates the compile class path:

cat resolution/org.slf4j/slf4j-api/1.7.36 resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 | sort -u | paste --delimiter=':' --serial > work/compile-class-path

Finally, it compiles our project and generates its JAR file:

javac -d work/main -g --class-path @work/compile-class-path --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .

It works.

Conclusion

Our Makefile is now capable of properly handling transitive dependencies.

We are using the Apache Maven Artifact Resolver project to resolve our dependencies. It is encapsulated in a single Java file, the Resolver.java file. We run it using JEP 330.

And that's it for this series. We might revisit the Makefile subject in future posts.

You can find the source code of the examples in this GitHub repository.