Learning Makefiles as a Java developer. Part 4: handling external dependencies

Marcio Endo

December 31, 2023

edit 2024-01-14: add a link to part 5 of the series

In the previous post we grew our project from a single class to multiple classes in a single package. In the process we have learned about empty targets and the $^ automatic variable.

The goal was to bring our project closer to the ones you would find in a professional setting. And, in the process, to learn about Makefiles and how to use them to build Java projects.

In this fourth part of this series we keep heading towards that goal. We will learn how to handle Java external dependencies in our project.

Let's continue.

Iteration 10: declaring an external dependency

Let's say that, instead of printing messages directly to System.out, we want to do it via a Logger. For this reason, we want to use SLF4J in our library.

We could simply download the JAR file from the Internet and use it.

But we want to have it automated somehow. In other words, we want to:

declare our dependency; and
have the JAR file downloaded automatically.

How can we do it in our project?

Adding `Logger` to our classes

As our focus in the build, the changes to our source file are not that important.

The important thing to notice is that the build now fails:

$ make
javac -d work/main -g --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
main/objectos/library/Hello.java:18: error: package org.slf4j does not exist
import org.slf4j.Logger;
                ^
main/objectos/library/Hello.java:19: error: package org.slf4j does not exist
import org.slf4j.LoggerFactory;
                ^
main/objectos/library/Hello.java:22: error: cannot find symbol
  private static final Logger LOGGER = LoggerFactory.getLogger(Hello.class);
                       ^
  symbol:   class Logger
  location: class Hello
main/objectos/library/Hello.java:22: error: cannot find symbol
  private static final Logger LOGGER = LoggerFactory.getLogger(Hello.class);
                                       ^
  symbol:   variable LoggerFactory
  location: class Hello
4 errors

As we do not have the SLF4J JAR file in our class path.

Declaring our dependency

In a Maven project we would add the following to the <dependencies> section:

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.7.36</version>
</dependency>

But we are not in a Maven project. Still, it shows that we do need to supply the GAV (group + artifact + version) coordinate.

Let's add the following to the top of our Makefile:

#
# objectos.library
#

## compile scope deps
COMPILE_DEPS = org.slf4j/slf4j-api/1.7.36

Our COMPILE_DEPS variable holds all of our declared dependencies. And the GAV coordinates are declared in the form:

$(GROUP_ID)/$(ARTIFACT_ID)/$(VERSION)

In other words, each of the G + A + V values are joined by the '/' character in order.

Defining the local repository and its layout

We will use the same layout of the Maven local repository.

In other words, the JAR file of the following GAV coordinate:

org.slf4j/slf4j-api/1.7.36

Will be stored at the following location:

$(LOCAL_REPO_PATH)/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

Where the LOCAL_REPO_PATH is a Makefile variable. In our Makefile it is declared like so:

#
# deps variables and functions
#

## local repository directory
LOCAL_REPO_PATH = repository

Our local repository is the repository directory under our current path.

Of course we could have defined with a different value, for example:

LOCAL_REPO_PATH = $(HOME)/.m2/repository

But, for the purposes of this blog post, a directory relative to our current path is a better option. It will make it easier to list its contents and clean when necessary.

Translating the dependency to a local JAR file

We haven't decided how we will download the dependency JAR file yet. But, for now, let's assume we have downloaded the JAR file and it is in our local repository.

We need the full path of the JAR file so we can properly specify the class-path during compilation:

javac --class-path [our dependency jar file goes here]

So we need to get from the following value:

org.slf4j/slf4j-api/1.7.36

To the following value:

$(LOCAL_REPO_PATH)/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

We need a function that maps GAVs to local repository paths.

As it turns out, Make allows us to define and call functions.

We will use them to translate the dependency declaration to its full path.

Iteration 11: a function to split a string around a separator

We declare our first Make function called split:

split = $(word $(3),$(subst $(2), ,$(1)))

It splits a string around a separator and returns the nth word.

Let's see it in action. We define the following temporary rule:

GAV = org.slf4j/slf4j-api/1.7.36

test-function:
	@echo GROUP_ID=$(call split,$(GAV),/,1)
	@echo ARTIFACT_ID=$(call split,$(GAV),/,2)
	@echo VERSION=$(call split,$(GAV),/,3)

And, when executed, it prints:

$ make print-GAV
GROUP_ID=org.slf4j
ARTIFACT_ID=slf4j-api
VERSION=1.7.36

Next, let's take a closer look at our function.

The `call` built-in function

To invoke our function we use the call built-in function:

$(call split,[string],[separator],[index])

The first argument is the name of our function, split in our case.

The following arguments are stored in temporary variables and serve as parameters of the invoked function.

Notice the $(1), $(2) and $(3) variables in our function definition:

split = $(word $(3),$(subst $(2), ,$(1)))

So when the call function evaluates our split function it assigns, in order, its arguments to the variables:

the name of the function, split, is assigned to the variable $(0);
the string that we want to split is assigned to the variable $(1);
the separator is assigned to the variable $(2); and
the index is assigned to the variable $(3).

So a call like so:

$(call split,foo/bar,/,2)

Is equivalent to:

$(word 2,$(subst /, ,foo/bar))

Next, let's explore the subst built-in function.

The `subst` built-in function

The syntax for the subst function is the following:

$(subst [from],[to],[text])

Which would be roughly equivalent to the following Java code:

String from = ...
String to = ...
String text = ...
String result = text.replace(from, to);

So the following invocation:

$(subst /, ,foo/bar)

Results in the words foo and bar.

The `word` built-in function

The syntax for the word function is the following:

$(word n,text)

It returns the nth word from text. The n index is 1-based. So the following:

$(word 2,apple orange banana)

Returns orange.

Iteration 12: defining the compilation class-path

Next we will update the "compile" section of our Makefile.

Converting the dependency GAV into a local repository file path

Let's declare our second Make function:

gav-local-repo1 = $(subst .,/,$(1))/$(2)/$(3)/$(2)-$(3).jar
gav-local-repo0 = $(call gav-local-repo1,$(call split,$(1),/,1),$(call split,$(1),/,2),$(call split,$(1),/,3))
gav-local-repo = $(LOCAL_REPO_PATH)/$(call gav-local-repo0,$(1))

OK, it is not a single function. It is a series of inter-related functions instead.

The main one, gav-local-repo, converts a single GAV into the corresponding local repository path. It uses the split function we defined in the previous section.

Let's see it in action. We define the following temporary rule:

GAV = org.slf4j/slf4j-api/1.7.36

test-function:
	@echo JAR=$(call gav-local-repo,$(GAV))

When executed it prints:

$ make test-function
JAR=repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

It works.

Converting a list of dependency GAVs: the `foreach` built-in function

We currently have a single dependency. However, we should be able to handle a list of dependencies.

So let's introduce the following gavs-to-local function that builds on top of the one we created in the previous section:

gavs-to-local = $(foreach gav,$(1),$(call gav-local-repo,$(gav)))

It uses the foreach built-in function.

So, for each gav in the specified list, it calls the gav-local-repo function.

Let's suppose we have a list of dependencies:

DEPS  = org.slf4j/slf4j-api/1.7.36
DEPS += com.example/some-library/1.2.3

Let's test our gavs-to-local function with the following temporary rule:

test-gavs-to-local:
	@echo $(call gavs-to-local,$(DEPS))

When executed it prints:

$ make test-gavs-to-local
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar repository/com/example/some-library/1.2.3/some-library-1.2.3.jar

It works.

Creating the class-path value

The following functions builds a class path suitable for using with the javac tool:

class-path = $(subst $(space),$(CLASS_PATH_SEPARATOR),$(1))

It replaces all the space characters in a list with the value of the CLASS_PATH_SEPARATOR variable.

The space variable is given by:

empty =
space = $(empty) $(empty)

Which is required when used as the first argument to the subst function. This method is documented in the GNU Make manual.

The CLASS_PATH_SEPARATOR is declared as:

CLASS_PATH_SEPARATOR = :

So, to be precise, the class path value is suitable for Linux and MacOS machines. We can declare different values to the variable depending on the operating system Make is running on. But this is currently beyond the scope of this blog post.

Let's see our class-path function in action. We declare the following test variables and rule:

DEPS  = org.slf4j/slf4j-api/1.7.36
DEPS += com.example/some-library/1.2.3

JARS = $(call gavs-to-local,$(DEPS))

test-class-path:
	@echo $(call class-path,$(JARS))

When executed it prints:

$ make test-class-path
repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:repository/com/example/some-library/1.2.3/some-library-1.2.3.jar

It works.

Updating the compilation rule

Finally, let's update the "compile" section of our Makefile.

We add the following variables:

## compile deps as local repo jars
COMPILE_DEPS_JARS = $(call gavs-to-local,$(COMPILE_DEPS))

## class path variable
COMPILE_CLASS_PATH = $(call class-path,$(COMPILE_DEPS_JARS))

And we alter the JAVACX variable to be the following:

## javac command options
JAVACX  = javac
JAVACX += -d $(CLASS_OUTPUT)
JAVACX += -g
JAVACX += --class-path $(COMPILE_CLASS_PATH)
JAVACX += --source-path $(MAIN)
JAVACX += $(SOURCES)

The changes are:

added the --class-path option to the javac command; and
changed the files to compile from the $^ automatic variables to the SOURCES variable.

Finally, we have to say that the compilation requires the dependency jars. So we add them as a prerequisite to our compilation rule, like so:

$(COMPILE_MARKER): $(SOURCES) $(COMPILE_DEPS_JARS)
	$(JAVACX)
	@touch $@

We have added the COMPILE_DEPS_JARS as an additional prerequisite of the COMPILE_MARKER target.

With these changes, let's try to execute our build:

$ make clean
rm -rf work

$ make
make: *** No rule to make target
	'repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar',
	needed by 'work/compile-marker'.  Stop.

The build failed. This is expected: we need to download the dependency from a central repository.

Iteration 13: fetching the external dependency

We now need to download the JAR file of our dependency into our local repository.

Remote repository location and layout

We will use Maven central as our remote repository. So we declare the following in our Makefile:

## remote repository location
REMOTE_REPO_URL = https://repo.maven.apache.org/maven2

The layout of the remote repository is the same as of our local repository.

In other words, if we need the following local repository file:

$(LOCAL_REPO_PATH)/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

We need to fetch the following remote repository file:

$(REMOTE_REPO_URL)/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar

Two features of Make will fit nicely with our current use-case:

pattern rules; and
substitution references.

A rule to download remote repository files

To download the JAR files from the remote repository, we add the following rule:

## remote repo wget command
REMOTE_REPO_WGETX  = wget
REMOTE_REPO_WGETX += --directory-prefix=$(LOCAL_REPO_PATH)
REMOTE_REPO_WGETX += --force-directories
REMOTE_REPO_WGETX += --no-host-directories
REMOTE_REPO_WGETX += --cut-dirs=1
REMOTE_REPO_WGETX += --no-verbose

## download dependency rule:
$(LOCAL_REPO_PATH)/%.jar:	
	$(REMOTE_REPO_WGETX) $(@:$(LOCAL_REPO_PATH)/%.jar=$(REMOTE_REPO_URL)/%.jar)

Let's break it down.

First, let's look at the target:

$(LOCAL_REPO_PATH)/%.jar:

The target uses a pattern as denoted by the '%' character. The target matches any file in the LOCAL_REPO_PATH directory having the jar extension.

Next, let's look at the recipe.

$(REMOTE_REPO_WGETX) $(@:$(LOCAL_REPO_PATH)/%.jar=$(REMOTE_REPO_URL)/%.jar)

It is a wget command where the file to be download is given by the following expression:

$(@:$(LOCAL_REPO_PATH)/%.jar=$(REMOTE_REPO_URL)/%.jar)

What it does is the following:

takes the target name, i.e, the value of the $@ automatic variable;
verifies if it matches the pattern $(LOCAL_REPO_PATH)/%.jar; and
if it matches, replaces the value with the pattern $(REMOTE_REPO_URL)/%.jar.

Putting it all together

Let's test the current iteration of our Makefile.

$ make
wget --directory-prefix=repository --force-directories --no-host-directories --cut-dirs=1 --no-verbose https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar
2023-12-30 12:45:33 URL:https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar [41125/41125] -> "repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar" [1]
javac -d work/main -g --class-path repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar --source-path main main/objectos/library/Hello.java main/objectos/library/Hi.java main/objectos/library/Say.java
jar --create --file=work/library.jar -C work/main .

And if we run it once again:

$ make
make: Nothing to be done for 'all'.

It works.

Conclusion

In this blog post we have shown a way to handle external dependencies in a "semi-automatic" manner.

It is "semi-automatic" because:

it downloads the JAR file automatically; but
it does not resolve nor download the transitive dependencies.

In the process we have discussed:

user defined functions;
the call built-in function;
the subst built-in function;
the word built-in function;
the foreach built-in function;
pattern rules; and
substitution references.

You can find the source code of the examples in this GitHub repository.

Continue reading

The fifth part of this series is already available.

You can continue reading by following this link.

Iteration 10: declaring an external dependency

Adding Logger to our classes

Declaring our dependency

Defining the local repository and its layout

Translating the dependency to a local JAR file

Iteration 11: a function to split a string around a separator

The call built-in function

The subst built-in function

The word built-in function

Iteration 12: defining the compilation class-path

Converting the dependency GAV into a local repository file path

Converting a list of dependency GAVs: the foreach built-in function

Creating the class-path value

Updating the compilation rule

Iteration 13: fetching the external dependency

Remote repository location and layout

A rule to download remote repository files

Putting it all together

Conclusion

Continue reading

Adding `Logger` to our classes

The `call` built-in function

The `subst` built-in function

The `word` built-in function

Converting a list of dependency GAVs: the `foreach` built-in function