A low-allocation template engine (Part 2). Objectos 0.4.4 released

Marcio Endo
March 5, 2023

Welcome to Objectos Weekly issue #016.

I have released Objectos 0.4.4. This is the last of the 0.4.x series, where I worked solely on the Objectos Code API. I will introduce Objectos HTML in the 0.5.x series next week. Please note, however, that Objectos Code should still be considered alpha quality.

In this issue I will continue to discuss how the Objectos Code engine works internally. Last issue was more about the API design. In this issue I will start to talk about the implementation.

Let's begin.

Our problem

In the last issue we came up with the following API design:

class HelloWorld extends JavaTemplate {
  String pkgName = "com.example";
  String name = "HelloWorld";

  @Override
  protected void definition() {
    packageDeclaration(pkgName);
    
    interfaceDeclaration(
      PUBLIC, name(name)
    );
  }
}

We hope the code above can be used to generate the following Java interface declaration:

package com.example;

public interface HelloWorld {}

The idea was to create an API that records and plays back method invocations.

The Recorder class

As we will have to record method invocations, let's call our class Recorder.

We want to use like the following:

var tmpl = new HelloWorld();

var recorder = new Recorder();

tmpl.accept(recorder);

And the JavaTemplate::accept method would be:

public abstract class JavaTemplate {
  Recorder recorder;

  public final void accept(Recorder recorder) {
    this.recorder = recorder;

    try {
      definition();
    } finally {
      this.recorder = null;
    }
  }
  
  (...)
}

So our Recorder class can intercept all of the method invocations happening in the definition method.

Recording the packageDeclaration method invocation

Let's implement the packageDeclaration method:

protected final void packageDeclaration(String name) {
  validatePackageName(name.toString());
  var recorder = recorder();
  int objectIndex = recorder.object(name);
  recorder.store(Element.PACKAGE_DECL, objectIndex);
}

A package declaration is always a top-level declaration. For this reason, this method does not return any value.

Let's look at the implementation in details.

Validating the package name

Objectos Code is a template engine. In other words, it does very little validation to user input. The reasoning is: the generated code will be compiled by a Java compiler; and the compiler is the ultimate validator.

Nevertheless, it will do some validation. It is to avoid things like:

@Override
protected void definition() {
  packageDeclaration(
  """
  com.example;
  // let's bypass the API
  public interface HelloWorld {}
  """
  );
}

So the first statement in the packageDeclaration implementation:

validatePackageName(name.toString());

Is responsible for checking if the name string contains only valid characters for a Java package name. If it contains an invalid character, such as a semicolon, it throws an IllegalArgumentException.

The name.toString() method invocation should return itself. Its sole purpose is to trigger an implicit null-check and take advantage of JEP 358.

Checking correct state

The next statement:

var recorder = recorder();

Is to make sure that the packageDeclaration method is being called inside a definition method.

Remember, the definition method is invoked by the accept method. It is only the latter that sets the recorder instance variable.

In other words, if packageDeclaration is not called inside a definition method, then recorder would be null.

Here's the recorder method implementation:

private Recorder recorder() {
  if (recorder == null) {
    throw new IllegalStateException(
      "Must be invoked inside a `definition` method"
    );
  }
  return recorder;
}

So if the recorder instance variable is null we assume something bad happened: for example, the packageDeclaration method might have been called outside a definition method.

Storing the "com.example" string

The next statement stores the com.example string:

int objectIndex = recorder.object(name);

Internally the Recorder class uses an Object[] array:

class Recorder {
  private Object[] objectArray;
  private int objectIndex; 
  ...
  public final int object(Object o) {
    ensureObjectArray();
    int result = objectIndex;
    objectArray[objectIndex++] = o;
    return result;
  }
  ...
}

The ensureObjectArray is responsible for initializing and resizing the array if necessary.

So the object method:

Storing the PACKAGE_DECL instruction

The last statement in the packageDeclaration method:

recorder.store(Element.PACKAGE_DECL, objectIndex);

Is responsible for the actual recording of the method invocation.

Where:

class Recorder {
  private int[] recorderArray;
  private int recorderIndex; 
  ...
  public final Element store(Element element, int v1) {
    ensureRecorderArray(2);
    recorderArray[recorderIndex++] = element.ordinal();
    recorderArray[recorderIndex++] = v1;
    return element;
  }
  ...
}

The ensureRecorderArray is responsible for initializing and resizing the array if necessary. It takes an int argument. The positive int value indicates that the array has to be large enough to hold that many additional values.

Then, the ordinal value of the enum constant is stored in the array. Along with the v1 value.

Our "memory" snapshot

So, in memory, we currently have the following:

object array: Object[]

idx |    value
--------------------
 0  | "com.example"

recorder array: int[]

idx |    value
--------------------
 0  | PACKAGE_DECL
 1  |      0

Ok, let's record the name method invocation next.

Recording the name method invocation

The name implementation is similar to the previous one:

protected final SimpleNameMarker name(String name) {
  validateIdentifier(name.toString());
  var recorder = recorder();
  int objectIndex = recorder.object(name);
  return recorder.store(Element.SIMPLE_NAME, objectIndex);
}

The difference is that it returns an instance of the SimpleNameMarker interface.

The purpose of the interface is to let the compiler know that the following invocation is valid:

interfaceDeclaration(PUBLIC, name(name))

In other words, the name method can be used as an argument to the interfaceDeclaration method.

Next, let's look at the differences from this method to the packageDeclaration method in details.

Validating the identifier

The first statement in the method is responsible for validating the identifier:

validateIdentifier(name.toString());

To be exact, it checks if the specified string contains invalid characters.

So, while it will reject values such as:

It will allow:

So, return does not contain any invalid characters. But it cannot be used as an identifier.

As mentioned, Objectos Code will do very little input validation; the Java compiler is the ultimate validator.

Storing the SIMPLE_NAME instruction

Let's jump right to the last statement:

return recorder.store(Element.SIMPLE_NAME, objectIndex);

Remember that:

The Element enum class is a non-exported (internal) type:

public enum Element implements InterfaceDeclarationMarker, SimpleNameMarker {
  INTERFACE_DECL,
  MODIFIER,
  PACKAGE_DECL,
  SIMPLE_NAME
  ... more
  ;
}

The SimpleNameMarker type is a sealed marker interface whose sole implementation is the Element enum class.

So the name method returns the same SimpleNameMarker instance each time is invoked.

In other words, it never allocates a new SimpleNameMarker instance.

Our "memory" snapshot

So our "memory/heap snapshot" currently looks like the following:

object array: Object[]

idx |    value
--------------------
 0  | "com.example"
 1  | "HelloWorld"

recorder array: int[]

idx |    value
--------------------
 0  | PACKAGE_DECL
 1  |      0  
 2  |  IDENTIFIER
 3  |      1

Recording the interfaceDeclaration method invocation

In the next issue we will look at the interfaceDeclaration method.

It takes an arbitrary number of arguments. Therefore its implementation is quite different than the two methods we saw earlier.

Stay tuned.

Arrays vs. an object graph

A reasonable question that might arise: aren't we still allocating to record the method invocations?

Well, we can't escape allocating.

But instead of using an object graph we are using just a few arrays. Which, perhaps ironically, is an object graph itself.

Objectos 0.4.4 released

You can read the release notes here.

There was a substantial work done on the documentation. Though it is still far from being complete.

As for the API, the following Objectos Code:

import objectos.code.ClassTypeName;
import objectos.code.JavaTemplate;
import objectos.code.TypeVariableName;

public class Box extends JavaTemplate {
  static final ClassTypeName OVERRIDE = ClassTypeName.of(Override.class);

  static final ClassTypeName STRING = ClassTypeName.of(String.class);

  static final TypeVariableName T = TypeVariableName.of("T");

  @Override
  protected final void definition() {
    autoImports();

    classDeclaration(
      PUBLIC, name("Box"),
      typeParameter("T"),

      field(
        PRIVATE, FINAL, T, name("value")
      ),

      constructor(
        PUBLIC,
        parameter(T, name("value")),

        p(THIS, n("value"), IS, n("value"))
      ),

      method(
        PUBLIC, T, name("get"),

        p(RETURN, n("value"))
      ),

      method(
        annotation(OVERRIDE),
        PUBLIC, FINAL, STRING, name("toString"),

        p(IF, condition(n("value"), EQ, NULL), block(
          p(RETURN, s("null"))
        ), ELSE, block(
          p(RETURN, n("value"), v("toString"))
        ))
      )
    );
  }
}

Generates the following Java code:

public class Box<T> {
  private final T value;

  public Box(T value) {
    this.value = value;
  }

  public T get() {
    return value;
  }

  @Override
  public final String toString() {
    if (value == null) {
      return "null";
    } else {
      return value.toString();
    }
  }
}

Until the next issue of Objectos Weekly

So that's it for today. I hope you enjoyed reading.

Please send me an e-mail if you have comments, questions or corrections regarding this post.