Learning Makefiles as a Java developer. Part 5: resolution files

Marcio Endo
January 7, 2024

edit 2024-01-14: add a link to part 6 of the series

Our project currently declares a single dependency: the SLF4J API library.

This particular dependency declares no dependencies of its own. When our project depends on it our only task is to install its JAR file locally. This is what we have done in the previous post of the series.

Some dependencies, on the other hand, may have dependencies of their own.

Take, for example, the Jackson Databind library. It depends on two other libraries:

And these dependencies might have dependencies of their own. In this particular case, they do not.

Therefore, if our project were to depend on Jackson Databind, we would require 3 JAR files:

We know this because the project README says so. We need a way to do this automatically.

We will start to design a solution in the fifth part of our blog post series.

Let's continue.

Iteration #14: the need to resolve transitive dependencies

Let's make a change to our project so it depends on the Jackson Databind library.

Quick reminder: our project is of no practical use; it is just a substrate for our Makefile learning.

The Hi.java file

Let's change our Hi class to the following:

import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.StringWriter;
import java.io.UncheckedIOException;

final class Hi {
  private static final ObjectMapper MAPPER = new ObjectMapper();
  
  static {
    try {
      // just to trigger a class load from the dependency
      MAPPER.createGenerator(new StringWriter());
    } catch (IOException e) {
      throw new UncheckedIOException(e);
    }
  }

  public void to(String who) {
    System.out.println("Hi " + who + "!");
  }
}

The important thing to notice is the import section of the file. The only direct dependencies of the class are:

And, of course, classes from the java.lang package.

Declaring our new dependency

Let's update our Makefile:

## compile scope deps
COMPILE_DEPS  = org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-databind/2.16.1

We have declared the Jackson Databind as a direct compile dependency.

Building our project

Let's try to build our project:

$ make

And here's the console output:

wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
2024-01-06 09:36:23 URL:https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar [1637611/1637611] -> "repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar" [1]
javac -d work/main -g --class-path repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
main/objectos/library/Hi.java:29: error: cannot access Versioned
      MAPPER.createGenerator(new StringWriter());
            ^
  class file for com.fasterxml.jackson.core.Versioned not found
1 error
make: *** [Makefile:118: work/compile-marker] Error 1

Notice the following:

Still, compilation failed. Our code indirectly depends on a class from the com.fasterxml.jackson.core package. In other words, we are missing the transitive dependencies.

Fixing our build

Let's apply quick fix to our project so it builds correctly:

## compile scope deps
COMPILE_DEPS  = org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-databind/2.16.1
## quick fix
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-annotations/2.16.1
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-core/2.16.1

This is a temporary workaround. We have declared the transitive dependencies as direct dependencies.

Invoking Make yields the following console output:

wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
2024-01-06 09:37:44 URL:https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar [78480/78480] -> "repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar" [1]
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
2024-01-06 09:37:44 URL:https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar [578125/578125] -> "repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar" [1]
javac -d work/main -g --class-path repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .

It compiles once again.

Iteration #15: externalizing the compile class path

The solution given in the previous section was just a workaround.

Let's start to design a complete solution.

We will begin with the class path of our compilation step.

Class path must be in a file

When a dependency is added to our project, the class path of the compilation step changes.

When the class path changes, the project must be recompiled.

Therefore, and ideally, we should express the compilation step like so:

$(COMPILE_MARKER): $(SOURCES) $(COMPILE_CLASS_PATH)
    $(JAVACX)
    @touch $@

So the project must be recompiled if:

For this rule to work, however, the COMPILE_CLASS_PATH must refer to a file.

And it just so happens that the javac tools allows for defining options in an external file.

Command-Line argument files

The javac tools allows for command-line argument files. They are regular text files containing arguments to the javac tool.

Say you have a text file named javac-opts with the following contents:

-d work/main
-g
--class-path json.jar:logging.jar

If we now invoke javac like so:

$ javac @javac-opts src/main/java/*.java

It will read the options from the javac-opts file. In other words, it would be equivalent to the following invocation:

$ javac -d work/main -g --class-path json.jar:logging.jar src/main/java/*.java

Let's use this feature with our compilation class path.

Saving the compilation class path in a file

Our compilation class path is currently a string:

## class path variable
COMPILE_CLASS_PATH = $(call class-path,$(COMPILE_DEPS_JARS))

Whose contents is computed via the class-path function.

We want it to be a regular file. Let's store it under our work directory:

## class path file
COMPILE_CLASS_PATH = $(WORK)/compile-class-path

And we create it using the same class-path function:

$(COMPILE_CLASS_PATH): Makefile
	echo $(call class-path,$(COMPILE_DEPS_JARS)) > $@

The class path file should be remade whenever a dependency is added, removed or updated. So we have added the Makefile file as a prerequisite. It is not the best of the prerequisites but it is enough for now; we do not want to declare our dependencies in a separate file.

Next, we must tell the javac command to read the class path from our file:

## javac command options
JAVACX  = javac
JAVACX += -d $(CLASS_OUTPUT)
JAVACX += -g
JAVACX += --class-path @$(COMPILE_CLASS_PATH)
JAVACX += --source-path $(MAIN)
JAVACX += $(SOURCES)

The javac command now uses a command-line argument file. More specifically, the --class-path value is given by contents of the COMPILE_CLASS_PATH file.

Putting it all together

Our updated "compile" section is the following:

#
# compile
#

## main source directory
MAIN = main

## all of our source files
SOURCES  = $(MAIN)/objectos/library/Hello.java
SOURCES += $(MAIN)/objectos/library/Hi.java
SOURCES += $(MAIN)/objectos/library/Say.java

## directory for the (main) compiled class files
CLASS_OUTPUT = $(WORK)/main

## compile deps as local repo jars
COMPILE_DEPS_JARS = $(call gavs-to-local,$(COMPILE_DEPS))

## class path variable
COMPILE_CLASS_PATH = $(WORK)/compile-class-path

## compilation marker
COMPILE_MARKER = $(WORK)/compile-marker

## javac command options
JAVACX  = javac
JAVACX += -d $(CLASS_OUTPUT)
JAVACX += -g
JAVACX += --class-path @$(COMPILE_CLASS_PATH)
JAVACX += --source-path $(MAIN)
JAVACX += $(SOURCES)

$(COMPILE_CLASS_PATH): Makefile
	echo $(call class-path,$(COMPILE_DEPS_JARS)) > $@

$(COMPILE_MARKER): $(SOURCES) $(COMPILE_CLASS_PATH)
	$(JAVACX)
	@touch $@

Let's test it by invoking Make. Here's the console output:

echo repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar > work/compile-class-path
javac -d work/main -g --class-path @work/compile-class-path --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .

It works.

Iteration #16: resolution files

We still do not know how we will resolve the transitive dependencies.

What we do know is that, for each of the declared direct dependencies:

## compile scope deps
COMPILE_DEPS  = org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += com.fasterxml.jackson.core/jackson-databind/2.16.1

We must find the list of all of the dependencies, direct or transitive. And, once again, this must be done for each of the declared direct dependencies.

So, for the SLF4J API, we need the following list:

repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

And for the Jackson Databind library, we need the following list:

repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar

Now, let's imagine each of those lists were in distinct text files.

We could concatenate the contents of both to build our compilation class path.

Resolution files

So let's create those files. We will call them "resolution files".

We will create the resolution/org.slf4j/slf4j-api/1.7.36 file. It contains all of the required JAR files for the SLF4J API library:

$ cat resolution/org.slf4j/slf4j-api/1.7.36
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

We will also create the resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 file. It contains all of the required JAR files for the Jackson Databind library:

$ cat resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar

Next, let's see if we can use these files to create a class path value.

Building the class path from the resolution files

The cat tool concatenates files and prints the result to the standard output:

$ cat resolution/org.slf4j/slf4j-api/1.7.36 \
	resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar

We have a list of all the JAR files.

This list might contain duplicates. While not strictly required, let's remove any duplicates using the sort tool:

$ cat resolution/org.slf4j/slf4j-api/1.7.36 \
	resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 \
	| sort -u 
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar
repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

Finally, let's concatenate all of lines separated by the ':' character. We will use the paste tool for this purpose:

$ cat resolution/org.slf4j/slf4j-api/1.7.36 \
	resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 \
	| sort -u | paste --delimiter=':' --serial
repository/com/fasterxml/jackson/core/jackson-annotations/2.16.1/jackson-annotations-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar:repository/com/fasterxml/jackson/core/jackson-databind/2.16.1/jackson-databind-2.16.1.jar:repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

So, given a list of resolution files, we can build our class path using:

cat [list of files] | sort -u | paste --delimiter=':' --serial

Next, let's use the resolution files in our Makefile.

Using the resolution files

We will update our Makefile to use the recently created resolution files.

First, our direct dependencies are now declared like so:

## compile scope deps
COMPILE_DEPS  = $(DEPS)/org.slf4j/slf4j-api/1.7.36
COMPILE_DEPS += $(DEPS)/com.fasterxml.jackson.core/jackson-databind/2.16.1

We are still using the same GAV format. But each direct dependency is prefixed with the $(DEPS) variable. The latter is declared like so:

## local resolution dir
DEPS = resolution

It is the directory where we stored our resolution files. So our COMPILE_DEPS is now a list of local files.

Next we update the $(COMPILE_CLASS_PATH) target to the following:

$(COMPILE_CLASS_PATH): $(COMPILE_DEPS) | $(WORK)
ifneq ($(COMPILE_DEPS),)
	cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@
else
	touch $@
endif

Before we break it down, let's see it in action.

First, we invoke make clean.

Then we invoke Make without arguments. Here's the console output:

mkdir work
cat resolution/org.slf4j/slf4j-api/1.7.36 resolution/com.fasterxml.jackson.core/jackson-databind/2.16.1 | sort -u | paste --delimiter=':' --serial > work/compile-class-path
javac -d work/main -g --class-path @work/compile-class-path --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .

It works.

Next, let's break it down.

New prerequisites

The first thing to notice is the new prerequisites of the compile class path target:

$(COMPILE_CLASS_PATH): $(COMPILE_DEPS) | $(WORK)

The compile class path should be remade whenever there's a change in any of the dependencies. Remember that the COMPILE_DEPS variable now refers to our resolution files.

The target will be remade when:

If a dependency is removed we will have to run a make clean prior to building our project.

Order-only prerequisites

The compile class path target introduces an order-only prerequisite. All prerequisites to the right of the | character are order-only:

$(COMPILE_CLASS_PATH): $(COMPILE_DEPS) | $(WORK)

The compile class path target creates a file under the work directory, namely the work/compile-class-path file. The work directory must exist before any file is created under it. The following target creates it:

$(WORK):
	mkdir $@

But the compile class path target must not be remade when work directory changes. In other words, we do not want to recreate the compile class path whenever a file is created, deleted or updated in the work directory. For this reason, we have declared the $(WORK) prerequisite as an order-only prerequisite.

The $^ automatic variable

Let's look at the recipe:

cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@

It uses the $^ automatic variable. This variable contains the names of all of the regular prerequisites. It does not include the names of the order-only prerequisites.

Conditional parts of a Makefile

Finally, the updated rule introduces a conditional directive:

ifneq ($(COMPILE_DEPS),)
	cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@
else
	touch $@
endif

The rule runs a different recipe depending on the contents of the COMPILE_DEPS variable.

If the COMPILE_DEPS variable is not equal to the empty string, then the target contents is created by concatenating all of the resolution files:

cat $^ | sort -u | paste --delimiter='$(CLASS_PATH_SEPARATOR)' --serial > $@

On the other hand, if the COMPILE_DEPS variable is the empty string, then the target is created without any contents:

touch $@

Conclusion

We have designed a solution for handling transitive dependencies in our Makefile. It is currently partially implemented.

It introduces the concept of resolution files. Each direct dependency declared in a Makefile should have a corresponding resolution file. A resolution file:

The compilation class path can be built by concatenating the contents of all of the resolution files.

The solution implementation is currently incomplete. The following is missing:

Completing the implementation of our solution will be the topic of the next and final blog post of this series.

You can find the source code of the examples in this GitHub repository.

Continue reading

The sixth part of this series is already available.

You can continue reading by following this link.