Contributing

Juicemacs builds upon other tools and frameworks and has its own code structure. For people with no prior experience of the dependencies, getting started can be difficult. This guide aims to ease the burden on new contributors and help them get familiar with the directory structure and the Truffle framework.

This guide assumes the following:

• The reader should have knowledge of Emacs Lisp, including its data types, language designs (i.e., Lisp-2, macros, lexical scoping, and dynamic scoping) and a bit of its internals (byte compilation and possibly native compilation).
• The reader should know Java and its basic syntax. In particular, there are a few new features from recent JDK versions that get used heavily in the codebase, including:
  • Markdown JavaDoc (JEP 467)
  • instanceof pattern matching (JEP 394)
  • switch pattern matching (JEP 441)
  • record classes (JEP 395)
• No prior knowledge of the Truffle framework is required. The reader is not expected to know how to work with Truffle when implementing simple Lisp functions (but it would certainly help if they do). When talking about Truffle caveats that do need a basic understanding of Truffle/Graal, this guide will include links to materials that I find useful when getting started with Truffle.

This guide will talk about the following:

• How Juicemacs uses code generation to ease developing:

  To keep in sync with GNU Emacs, Juicemacs heavily relies on code generation (see emacs-extractor). This will keep the Lisp function signatures (extracted as Java method signatures) and documentation (extracted into JavaDoc) remain synchronized. The next section, 3, will cover how to work with the generated code.

• What things you will need to take note of to make the Truffle JIT magic happen:

  Juicemacs uses the Truffle framework for a JIT compiling ELisp interpreter. This framework, however, requires quite a few workarounds to make the JIT magic happen. We will document these caveats in dedicated sections. (If you encounter any new gotchas, you are welcome to add them here!)

• What to work on

Like any language implementation, we need to implement a lot of built-in functions in Juicemacs. GNU Emacs has around 1800, and there's still a long long way to go. With code generation from emacs-extractor and the Truffle annotation processor, implementing a built-in Lisp subroutine is not that hard and can be a good way to get started with the codebase.

For example, we can get to implement the consp function in just two steps:

1. Locate the pre-generated placeholder implementation. In this case, the function is defined in data.c in GNU Emacs, and the generated placeholder can be found in BuiltInData.java:

   @ELispBuiltIn(name = "consp", minArgs = 1, maxArgs = 1)
   @GenerateNodeFactory
   public abstract static class FConsp extends ELispBuiltInBaseNode {
       @Specialization
       public static Void consp(Object object) {
           throw new UnsupportedOperationException();
       }
   }
   

2. Implement the function in two lines:

    @ELispBuiltIn(name = "consp", minArgs = 1, maxArgs = 1)
    @GenerateNodeFactory
    public abstract static class FConsp extends ELispBuiltInBaseNode {
        @Specialization
   -    public static Void consp(Object object) {
   -        throw new UnsupportedOperationException();
   +    public static boolean consp(Object object) {
   +        return object instanceof ELispCons;
        }
    }
   

Of course, most built-in functions will require more than two lines of code, but the above covers the gist of a built-in function definition in Juicemacs:

@ELispBuiltIn metadata

@ELispBuiltIn annotations contain basic function information, such as the function name and argument counts. They are used by the language context to register built-in functions. (See ELispBuiltIns.java for more implementation details.)

@GenerateNodeFactory

This annotation asks Truffle to generate factory classes for this function and gather all the factories into a list.

abstract static class FConsp extends ELispBuiltInBaseNode

This class definition is required for Truffle code generation to kick in:

• abstract: Truffle will generate child classes.
• ELispBuiltInBaseNode: This class extends the Node class from Truffle and contains several abstract executeXXX methods. Truffle will generate JIT-capable implementations for these executeXXX methods, which in turn will call @Specialization methods for the actual work.

@Specialization

This annotation marks the implementation of the consp function. A class can contains several @Specialization methods, each with different parameter types (and optionally different return types) but the same parameter count (similar to Java method overloading):

@ELispBuiltIn(name = "length", minArgs = 1, maxArgs = 1)
@GenerateNodeFactory
public abstract static class FLength extends ELispBuiltInBaseNode {
    @Specialization
    public static long lengthCons(ELispCons sequence) {
        return sequence.size();
    }
    @Specialization
    public static long lengthVector(ELispVectorLike<?> sequence) {
        return sequence.size();
    }
    @Specialization
    public static long lengthString(ELispString sequence) {
        return sequence.length();
    }
}

Truffle will generate code to dispatch function calls to correponding specializations.

Most of the above should be kept as is, because they will be overwritten when we rerun the code generation to update JavaDoc and incorporate GNU Emacs changes. Therefore, for the code generator to work correctly, there are a few things to note:

• You can only place extra method / class definitions either inside the function class, or at the top of the enclosing class (at the top of BuiltInData.java in this case).
• Do not modify the argument names, which are also used by the code generator and should already be in sync with the JavaDoc.

Sometimes the placeholder implementation is not yet generated. In that case, you will need to modify the emacs-extractor code to extract them from the GNU Emacs code.

While implementing Lisp functions is straightforward, making them run fast can be a challenge because it usually involve writing a few Truffle nodes. We will begin with a few examples:

One of the main challenges for working with Truffle is that it (and the Graal compiler) can be quite picky about the interpreter code. Truffle even has a method blocklist to prevent compilation of methods that are hard to partial-evaluate.

We have a few PMD rules to detect code that might cause trouble (see elisp/scripts/pmd-elisp-cautions.xml and elisp/scripts/pmd-truffle-practices.xml) and you may also find other pitfalls by trying to generating native images for Juicemacs.

This section tries to summarize these pitfalls.

• Recursive methods cause trouble because Truffle tries to inline everything, leading to exploding code size. Annotate them with @TruffleBoundary to disallow partial evaluation.
• Truffle uses VirtualFrame to track variables. When passing the frame instance to a method, the method should not be used in a loop, unless the loop is mark for @ExplodeLoop. (Otherwise the method will fail to compile.)
• When calling an polymorphic method (e.g., Object::toString), Truffle (during native image compilation) will mark all implementing methods for compilation.

The above leads to the following:

• Highly complicated classes (usually with high recursive methods) should only be used inside @TruffleBoundary functions. This includes BigInteger, HashMap and a thousand others.
• Locale related operations involves HashMap, and should be avoided. This means, all java.lang.String concatenations must also happen inside @TruffleBoundary because they somehow calls formatting methods which can be locale dependent at times.
• Working around string concatenation is painful, not only because we need a dedicated @TruffleBoundary utility method, but also because it forbids us from using any method (in external libraries or Java standard library) that does string concatenation in partial evaluated code.
• When calling polymorphic methods, any of them failing the compilation will make the whole compilation fail. Since most Java libraries (standard or external) uses java.lang.String for errors, this basically means: no polymorphic methods unless the interface is yours. When you see compilation errors from strange classes, it's very likely that it is from an unnoticed implementation of a polymorphic method.

There are more. Ways to discover them are:

• Try ./gradlew :app:nativeCompile.
• Enable engine.TraceCompilation and set engine.CompilationFailureAction to Diagnose. (See ELispLanguageTest.java.)

Using a Java debugger should be enough most of the time. See TestingUtils.java for some useful Truffle options when debugging:

• engine.Compilation: It seems sometimes Truffle can corrupt the stack info, making Java debuggers basically useless. If the bug is reproducible, you may try turning off compilation.
• inspect: Enable you to use Chrome debugger to debug Lisp code since we don't have edebug here. (It's not working very well though.)
• engine.CompilationFailureAction / engine.TraceCompilation: Useful when you are diagnosing performance issues.

• Places marked with UnsupportedOperationException or TODO comments:

  We mark out unimplemented features in the codebase with UnsupportedOperationException or TODO comments. And grepping for them are a good way to find things to work on.

• Failing conformance tests:

  All those el-semi-fuzz and ERT tests do reveal conformance issues in Juicemacs. Currently, Juicemacs does not pass all the ERT tests and occasionally fails el-semi-fuzz tests (because of the randomness). Looking into these failing cases can be a good way to learn about the internals of both Juicemacs and GNU Emacs.

• Adding more tests:

  There are quite a lot of ERT tests in the GNU Emacs codebase. However, currently we only run a few of them in our unit test (ELispRegressionTest.java). Getting more of them to run can be quite beneficial too.

• Performance engineering:

  There is a TODO list here at docs/TODO.html. The reason I chose to use a text file instead of issue trackers was because they were mostly about the internals of Juicemacs and might require some Truffle hacking, and there were simply too many things to track and learn about. There are still some unsolved issues (and one might add more), and you are welcome to look into them if you want to learn more about Truffle the framework.