Learning Makefiles as a Java developer. Part 5: resolution files
edit 2024-01-14: add a link to part 6 of the series
Our project currently declares a single dependency: the SLF4J API library.
This particular dependency declares no dependencies of its own. When our project depends on it our only task is to install its JAR file locally. This is what we have done in the previous post of the series.
Some dependencies, on the other hand, may have dependencies of their own.
Take, for example, the Jackson Databind library. It depends on two other libraries:
And these dependencies might have dependencies of their own. In this particular case, they do not.
Therefore, if our project were to depend on Jackson Databind, we would require 3 JAR files:
-
jackson-databind.jar
; -
jackson-annotations.jar
; and -
jackson-core.jar
.
We know this because the project README says so. We need a way to do this automatically.
We will start to design a solution in the fifth part of our blog post series.
Let's continue.
Iteration #14: the need to resolve transitive dependencies
Let's make a change to our project so it depends on the Jackson Databind library.
Quick reminder: our project is of no practical use; it is just a substrate for our Makefile learning.
The Hi.java
file
Let's change our Hi
class to the following:
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.StringWriter;
import java.io.UncheckedIOException;
final class Hi {
private static final ObjectMapper MAPPER = new ObjectMapper();
static {
try {
// just to trigger a class load from the dependency
MAPPER.createGenerator(new StringWriter());
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public void to(String who) {
System.out.println("Hi " + who + "!");
}
}
The important thing to notice is the import section of the file. The only direct dependencies of the class are:
-
the
ObjectMapper
class from Jackson Databind; and -
three classes from the JDK.
And, of course, classes from the java.lang
package.
Declaring our new dependency
Let's update our Makefile:
## compile scope deps
COMPILE_DEPS = org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-databind/2.16.1
We have declared the Jackson Databind as a direct compile dependency.
Building our project
Let's try to build our project:
$ make
And here's the console output:
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
2024-01-06 09:36:23 URL:https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar [1637611/1637611] -> "repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar" [1]
javac -d work/main -g --class-path repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
main/objectos/library/Hi.java:29: error: cannot access Versioned
MAPPER.createGenerator(new StringWriter());
^
class file for com.fasterxml.jackson.core.Versioned not found
1 error
make: *** [Makefile:118: work/compile-marker] Error 1
Notice the following:
-
the
jackson-databind
JAR file was correctly downloaded; and -
the
javac --class-path
option correctly included thejackson-databind
JAR file.
Still, compilation failed.
Our code indirectly depends on a class from the com.fasterxml.jackson.core
package.
In other words, we are missing the transitive dependencies.
Fixing our build
Let's apply quick fix to our project so it builds correctly:
## compile scope deps
COMPILE_DEPS = org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-databind/2.16.1
## quick fix
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-annotations/2.16.1
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-core/2.16.1
This is a temporary workaround. We have declared the transitive dependencies as direct dependencies.
Invoking Make yields the following console output:
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
2024-01-06 09:37:44 URL:https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar [78480/78480] -> "repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar" [1]
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
2024-01-06 09:37:44 URL:https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar [578125/578125] -> "repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar" [1]
javac -d work/main -g --class-path repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .
It compiles once again.
Iteration #15: externalizing the compile class path
The solution given in the previous section was just a workaround.
Let's start to design a complete solution.
We will begin with the class path of our compilation step.
Class path must be in a file
When a dependency is added to our project, the class path of the compilation step changes.
When the class path changes, the project must be recompiled.
Therefore, and ideally, we should express the compilation step like so:
$(COMPILE_MARKER): $(SOURCES) $(COMPILE_CLASS_PATH)
$(JAVACX)
@touch $@
So the project must be recompiled if:
-
any of the Java source files changes; or
-
the class path changes.
For this rule to work, however, the COMPILE_CLASS_PATH
must refer to a file.
And it just so happens that the javac
tools allows for defining options in an external file.
Command-Line argument files
The javac
tools allows for command-line argument files.
They are regular text files containing arguments to the javac
tool.
Say you have a text file named javac-opts
with the following contents:
-d work/main
-g
--class-path json.jar:logging.jar
If we now invoke javac
like so:
$ javac @javac-opts src/main/java/*.java
It will read the options from the javac-opts
file.
In other words, it would be equivalent to the following invocation:
$ javac -d work/main -g --class-path json.jar:logging.jar src/main/java/*.java
Let's use this feature with our compilation class path.
Saving the compilation class path in a file
Our compilation class path is currently a string:
## class path variable
COMPILE_CLASS_PATH = $(call class-path,$(COMPILE_DEPS_JARS))
Whose contents is computed via the class-path
function.
We want it to be a regular file. Let's store it under our work directory:
## class path file
COMPILE_CLASS_PATH = $(WORK)/compile-class-path
And we create it using the same class-path
function:
$(COMPILE_CLASS_PATH): Makefile
echo $(call class-path,$(COMPILE_DEPS_JARS)) > $@
The class path file should be remade whenever a dependency is added, removed or updated.
So we have added the Makefile
file as a prerequisite.
It is not the best of the prerequisites but it is enough for now;
we do not want to declare our dependencies in a separate file.
Next, we must tell the javac
command to read the class path from our file:
## javac command options
JAVACX = javac
JAVACX += -d $(CLASS_OUTPUT)
JAVACX += -g
JAVACX += --class-path @$(COMPILE_CLASS_PATH)
JAVACX += --source-path $(MAIN)
JAVACX += $(SOURCES)
The javac
command now uses a command-line argument file.
More specifically, the --class-path
value is given by contents of the COMPILE_CLASS_PATH
file.
Putting it all together
Our updated "compile" section is the following:
#
# compile
#
## main source directory
MAIN = main
## all of our source files
SOURCES = $(MAIN)/objectos/library/Hello.java
SOURCES += $(MAIN)/objectos/library/Hi.java
SOURCES += $(MAIN)/objectos/library/Say.java
## directory for the (main) compiled class files
CLASS_OUTPUT = $(WORK)/main
## compile deps as local repo jars
COMPILE_DEPS_JARS = $(call gavs-to-local,$(COMPILE_DEPS))
## class path variable
COMPILE_CLASS_PATH = $(WORK)/compile-class-path
## compilation marker
COMPILE_MARKER = $(WORK)/compile-marker
## javac command options
JAVACX = javac
JAVACX += -d $(CLASS_OUTPUT)
JAVACX += -g
JAVACX += --class-path @$(COMPILE_CLASS_PATH)
JAVACX += --source-path $(MAIN)
JAVACX += $(SOURCES)
$(COMPILE_CLASS_PATH): Makefile
echo $(call class-path,$(COMPILE_DEPS_JARS)) > $@
$(COMPILE_MARKER): $(SOURCES) $(COMPILE_CLASS_PATH)
$(JAVACX)
@touch $@
Let's test it by invoking Make. Here's the console output:
echo repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar > work/compile-class-path
javac -d work/main -g --class-path @work/compile-class-path --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .
It works.
Iteration #16: resolution files
We still do not know how we will resolve the transitive dependencies.
What we do know is that, for each of the declared direct dependencies:
## compile scope deps
COMPILE_DEPS = org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-databind/2.16.1
We must find the list of all of the dependencies, direct or transitive. And, once again, this must be done for each of the declared direct dependencies.
So, for the SLF4J API, we need the following list:
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
And for the Jackson Databind library, we need the following list:
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
Now, let's imagine each of those lists were in distinct text files.
We could concatenate the contents of both to build our compilation class path.
Resolution files
So let's create those files. We will call them "resolution files".
We will create the resolution/org.slf4j/slf4j-api/1.7.36
file.
It contains all of the required JAR files for the SLF4J API library:
$ cat resolution/org.slf4j/slf4j-api/1.7.36
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
We will also create the resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1
file.
It contains all of the required JAR files for the Jackson Databind library:
$ cat resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
Next, let's see if we can use these files to create a class path value.
Building the class path from the resolution files
The cat
tool concatenates files and prints the result to the standard output:
$ cat resolution/org.slf4j/slf4j-api/1.7.36 \
resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
We have a list of all the JAR files.
This list might contain duplicates.
While not strictly required, let's remove any duplicates using the sort
tool:
$ cat resolution/org.slf4j/slf4j-api/1.7.36 \
resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 \
| sort -u
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
Finally, let's concatenate all of lines separated by the ':' character.
We will use the paste
tool for this purpose:
$ cat resolution/org.slf4j/slf4j-api/1.7.36 \
resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 \
| sort -u | paste --delimiter=':' --serial
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar:repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
So, given a list of resolution files, we can build our class path using:
cat [list of files] | sort -u | paste --delimiter=':' --serial
Next, let's use the resolution files in our Makefile.
Using the resolution files
We will update our Makefile to use the recently created resolution files.
First, our direct dependencies are now declared like so:
## compile scope deps
COMPILE_DEPS = $(DEPS)/org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += $(DEPS)/com.fasterxml.jackson.core/jackson-databind/2.16.1
We are still using the same GAV format.
But each direct dependency is prefixed with the $(DEPS)
variable.
The latter is declared like so:
## local resolution dir
DEPS = resolution
It is the directory where we stored our resolution files.
So our COMPILE_DEPS
is now a list of local files.
Next we update the $(COMPILE_CLASS_PATH)
target to the following:
$(COMPILE_CLASS_PATH): $(COMPILE_DEPS) | $(WORK)
ifneq ($(COMPILE_DEPS),)
cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@
else
touch $@
endif
Before we break it down, let's see it in action.
First, we invoke make clean
.
Then we invoke Make without arguments. Here's the console output:
mkdir work
cat resolution/org.slf4j/slf4j-api/1.7.36 resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 | sort -u | paste --delimiter=':' --serial > work/compile-class-path
javac -d work/main -g --class-path @work/compile-class-path --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .
It works.
Next, let's break it down.
New prerequisites
The first thing to notice is the new prerequisites of the compile class path target:
$(COMPILE_CLASS_PATH): $(COMPILE_DEPS) | $(WORK)
The compile class path should be remade whenever there's a change in any of the dependencies.
Remember that the COMPILE_DEPS
variable now refers to our resolution files.
The target will be remade when:
-
a new resolution file is added; or
-
the contents of an existing resolution file is changed.
If a dependency is removed we will have to run a make clean
prior to building our project.
Order-only prerequisites
The compile class path target introduces an order-only prerequisite.
All prerequisites to the right of the |
character are order-only:
$(COMPILE_CLASS_PATH): $(COMPILE_DEPS) | $(WORK)
The compile class path target creates a file under the work
directory, namely the work/compile-class-path
file.
The work
directory must exist before any file is created under it.
The following target creates it:
$(WORK):
mkdir $@
But the compile class path target must not be remade when work directory changes.
In other words, we do not want to recreate the compile class path whenever a file is created, deleted or updated in the work
directory.
For this reason, we have declared the $(WORK)
prerequisite as an order-only prerequisite.
The $^
automatic variable
Let's look at the recipe:
cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@
It uses the $^
automatic variable.
This variable contains the names of all of the regular prerequisites.
It does not include the names of the order-only prerequisites.
Conditional parts of a Makefile
Finally, the updated rule introduces a conditional directive:
ifneq ($(COMPILE_DEPS),)
cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@
else
touch $@
endif
The rule runs a different recipe depending on the contents of the COMPILE_DEPS
variable.
If the COMPILE_DEPS
variable is not equal to the empty string, then the target contents is created by concatenating all of the resolution files:
cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@
On the other hand, if the COMPILE_DEPS
variable is the empty string, then the target is created without any contents:
touch $@
Conclusion
We have designed a solution for handling transitive dependencies in our Makefile. It is currently partially implemented.
It introduces the concept of resolution files. Each direct dependency declared in a Makefile should have a corresponding resolution file. A resolution file:
-
is a regular file under the
resolution
directory; -
has a path name of the form
[group id]/[artifact id]/[version]
; and -
contains the list of all of the JAR files required by the corresponding dependency.
The compilation class path can be built by concatenating the contents of all of the resolution files.
The solution implementation is currently incomplete. The following is missing:
-
automatically resolve the transitive dependencies; and
-
automatically download the transitive dependencies.
Completing the implementation of our solution will be the topic of the next and final blog post of this series.
You can find the source code of the examples in this GitHub repository.
Continue reading
The sixth part of this series is already available.
You can continue reading by following this link.