Notes To Myself

• Text Rendering Hates You - Aria Desires
• The Craft of Text Editing - Craig A. Finseth

Despite the usual complaints about how Electron bloats, I actually think it is good to simply embed the rendering engine of any browser:

1. They should support BIDI out of the box. (Well, since Emacs also supports turning off BIDI, this might not be as good after all.)
2. Decent performance.
3. Multi-media support.

Any of the above will take significant effort if we are to implement them ourselves (correctly). Of course we can offload the work to some UI frameworks, but I believe BIDI support will rule out most of the so-called modern GPU-powered ones.

I know little about the history behind Emacs. But according to XEmacs Lisp Reference Manual: 63. MULE and Frequently Asked Questions about Emacs - Question 27 (What variants of GNU Emacs exist?), MULE comes from Japanese forks of Emacs, which I do think are some Emacs variants (besides XEmacs) that should be remembered.

These days, Emacs can handle multilingual text and can (seemingly) losslessly convert between Unicode and other charsets, even for some charsets not yet mapped to Unicode. Emacs achieves this by extending the code point range of its internal UTF-8-like string representation to 0 - #x3FFFFF: The 0 - #x10FFFF range is for Unicode, and the remaining is for lossless storage of non-convertible characters.

The following example is inspired by Unicodeにマッピングされていない文字 - 清水 川のScrapbox ("Characters that are not yet mapped to Unicode" in English):

(let* (;; 洲﨑神社 if the text were convertible to Unicode
       (jis-encoded-bytes  "\x1b$B='yu?@<R\x1b(B")
       ;; Losslessly decoding to Emacs strings
       (lossless-string    (decode-coding-string jis-encoded-bytes 'iso-2022-jp))
       (convertible-char-1 (aref lossless-string 0))
       (non-convertible    (aref lossless-string 1))
       (re-encoded-bytes   (encode-coding-string lossless-string 'iso-2022-jp)))
  `(("Converted string"
     ;; Replace non-Unicode chars so that the outputs won't disrupt the coding of this file.
     ;; It did corrupted the encoding last time...
     ,(replace-regexp-in-string "[^\0-\U0010FFFF]" "?" lossless-string))
    ("Original bytes" ,(prin1-to-string jis-encoded-bytes))
    ("Re-encoded bytes" ,(prin1-to-string re-encoded-bytes))
    ("Convertible char" ,(format "%c (#x%06x < #x10FFFF)" convertible-char-1 convertible-char-1))
    ("Non-convertible"  ,(format "__ (#x%06x > #x10FFFF)" non-convertible))))

So awhile ago I started bootstraping files from emacs/lisp/language/*.el, and… there you go. Another example right in the source code:

(let ((non-unicode (with-temp-buffer
                     (insert-file (locate-file "ethiopic.el" load-path))
                     (replace-regexp "[\0-\U0010FFFF]+" "")
                     (buffer-string))))
  (format "#x%X > #x10FFFF" (aref non-unicode 0)))

#x1A01CA > #x10FFFF

There are some more files similar to ethiopic.el:

(defun file-contains-non-unicode? (file)
  (with-temp-buffer
    (insert-file file)
    (search-forward-regexp "[^\0-\U0010FFFF]" nil t)))

(let* ((language-dir (file-name-parent-directory (locate-file "ethiopic.el" load-path)))
       (files (append (directory-files language-dir t ".*\\.el$")
                      (directory-files (file-name-concat (file-name-parent-directory language-dir)
                                                         "international")
                                       t ".*\\.el$")))
       non-unicode-files)
  (dolist (file files)
    (if (file-contains-non-unicode? file)
        (push (list (file-name-nondirectory file)) non-unicode-files)))
  non-unicode-files)

Most Truffle languages store local variables on frames (VirtualFrame). There are some optimizations with this approach:

• Truffle can optimize many things for you by inlining and placing things on a real stack after JIT-compilation.
• Nodes for reading and setting variables can cache the location of a specific variable, yielding better performance.
• Lexical scopes are now materialized frames, which Truffle is aware and can potentially optimize/inline away the cost.

However, as always, Emacs Lisp brings some problems:

• Most languages use static analysis to map variable names to their slot number. Static analysis ensures the slot numbers do not change between calls and is safe to cache.

  function f(
    a,  // slot#0
    b,  // slot#1
  ) {
    const c = 0;  // slot#2
    if (a) {
      let d = 1;  // slot#3
      b += d + a;
    }
    let e = c + b;  // slot#4 or slot#3 if reusing slots
    return e;
  }
  

• As for ELisp, well, you cannot do static analysis with a dynamic AST.

  (defun f (a  ;; slot#0
            b  ;; slot#1
            )
    (some-macro
     ;; the slot of `c' depends on the expansion result of `some-macro'
     (let ((c 1)))))
  

  Alternatively, similar to how we handle things in ELispInterpretedNode.java, we may dynamically assign slot numbers and cache at runtime instead. However, Truffle FrameDescriptor requires a constant slot count at declaration time, which poses yet another challenge.

var absFuncRootNode = new FunctionRootNode(
  this,
  AbsFunctionBodyExprNodeGen.create(new ReadFunctionArgExprNode(0)),
);
context.globalScopeObject.newConstant(
  "Math.abs",
  new FunctionObject(absFuncRootNode.getCallTarget()),
);

Basically:

• FunctionCallExprNode evaluates all its children and dispatches the call with FunctionDispatchNode, which uses a DirectCallNode to call a FunctionObject.
• Math.abs is a FunctionObject which is simply a wrapper for a CallTarget,
• The CallTarget is obtained from a RootNode subclass (FunctionRootNode), which is wrapped around a function body node (AbsFunctionBodyExprNode extending EasyScriptExprNode).
• AbsFunctionBodyExprNode uses ReadFunctionArgExprNode as a child node to read parameters from the VirtualFrame.
• @GenerateNodeFactory is used to ease writing more built-in functions.

So it seems I have mis-implemented closure creation: (cl-loop for i from 0 to 10 collect (lambda () i)) should let all these lambdas share their AST and internal FunctionRootNode so as to reduce costs. Otherwise, because nodes should only be created in interpreted mode, every time a new closure is created, the code will deopt, jumping from compiled code to interpreted execution, causing significant slowdown.

I will try to follow how GraalJs handles this. (TL;DR: Read JSArguments.java, JSFunctionObject.java and finally JSFunctionCallNode.java.)

1. I was benchmarking (mapc ...), so let's take a look at Array.forEach in GraalJs, which seems to be in ArrayPrototypeBuiltins.java.
2. The corresponding node implementation is JSArrayForEachNode, which depends on two other nodes:
   • ArrayForEachIndexCallOperation: Parent node
   • MaybeResultNode: Whether to continue execution, used by ArrayForEachIndexCallOperation node
3. After basic type checking, the @Specialization method in JSArrayForEachNode eventually calls forEachIndexCall, which is implemented by ArrayForEachIndexCallOperation.
4. ArrayForEachIndexCallOperation seems a tiny wrapper around ForEachIndexCallNode, located in ForEachIndexCallNode.java (which also contains the definition of MaybeResult).
5. ForEachIndexCallNode:
   • executeForEachIndexFast
   • callback(index, value, target, callback, callbackThisArg, currentResult)
   • callbackNode.apply(index, value, target, callback, callbackThisArg, currentResult)
   • maybeResultNode.apply(index, value, callbackResult, currentResult)
6. callbackNode is a DefaultCallbackNode passed from ArrayForEachIndexCallOperation.
7. The callbackNode further wraps around JSFunctionCallNode and JSArguments: callNode.executeCall(JSArguments.create(callbackThisArg, callback, value, boxIndex(index), target)).

Looking at JSArguments and JSFunctionObject, we can take a glimpse of how GraalJs handles closures:

• Functions created from a certain "closure creating node" share their RootNode.
• JS Functions, when called, are passed two extra arguments: this and the function object itself.
• Function objects store captured parent frames.

So, in order to follow suit, we need to:

1. Change the current call convention
2. Let function special form (and make-closure function) caches and reuse previous AST (RootNode).

Its approach is quite similar to the previous tutorial:

• SLInvokeNode corresponds to function call AST node. It uses InteropLibrary to dispatch calls though.
• SLFunctionRegistry: Maps function names to SLFunction objects.
• SLFunction wraps a RootCallTarget.
• SLPrintlnBuiltin extends SLBuiltinNode (which has the @GenerateNodeFactory annotation and extends SLExpressionNode).
• Built-in functions are registered by SLContext::installBuiltins, which in turn calls SLLanguage::lookupBuiltin to setup SLReadArgumentNode and functions.
• Notably, it seems to use a CyclicAssumption to detect call target changes.

• JSFunctionCallNode has a internal function object cache.
• No @GenerateNodeFactory is used. Global functions are setup with JSRealm::setupGlobals with hand-written function lists.
• JSFunction wraps (deeply) a CallTarget.
• Notably, JSFunctionCallNode implements a rather complex caching logic in its executeAndSpecialize function.

• The built-in functions in GraalPython show-case an advanced (undocumented?) usage of @GenerateNodeFactory:

  It seems that, when annotating inner classes with @GenerateNodeFactory, the DSL processor will also generate a factory for the outer class, containing a getFactories() method returning all the inner factories. This can be extremely convenient for writing and loading built-in functions in batch.

• EmacsWiki: Hacker Guide
• Emacs Types - remacs Wiki
• emacs 源码分析（二）: Contains a pretty diagram for some structs
• Emacs internal string encoding - remacs Issue #499
• read0 @ src/lread.c: The Emacs Lisp S-expr lexer & parser
• [PDF] Conceptual Views of EMACS's Architecture - chrismennie.ca!
• bytecode.rs @ rune
• [PDF] GNU Emacs Lisp Bytecode Reference Manual (TeX source here)

• discussion about display engine and GUI model of emacs - rune Issue #61
• The Craft of Text Editing –or– Emacs for the Modern World