TODO

Currently thread-local buffer-locals are treated as thread-locals. But they should be buffer-local thread-locals ;-) Buffalo buffalo buffalo.

So, other than buffer-locals (and our thread-locals), Emacs also has "frame-locals" and "terminal-locals". I think we might need to unify the API a bit to make things easier to implement. Currently, we do have this ValueStorage.Value interface though, so maybe:

• ThreadLocalStorage: We treat this specially now, because probably we could use a bit of performance for them.
• Value interface for locals:
  • ForwardedPerBuffer
  • BufferLocal
  • ForwardedPerFrame?
  • ForwardedPerTerminal?

Lazy data for concat, useful for nested concat/format/... calls.

format function calls very often comes with constant format strings. It might help if we can turn (format "example: %d" var) into (concat "example" (format-%d var)) during JIT.

PCRE-sljit seems to use a technique called "static backtracking". The basic idea is to represent success/failures with control flows instead of boolean values, which is only achievable under JIT with unstructured control flow. (For example, the patten /p/ checks the current char, if it matches p, it jumps to some ACCEPT path or otherwise a BACKTRACK path.)

I tried to re-implement the engine this way, but find the performance quite bad. One explanation may be: Graal/Truffle does not have very good support for unstructured control flow. Quoting from GraalVM Slack:

Josef Haider (on Dec 2019): MERGE_EXPLODE supports unstructured control flow by falling back to an explicit state machine in the compiled code.

Chris Seaton (on Oct 2022): I wrote a blog post about irreducible loops, and yes Graal only supports them with a custom duplication phase, purely for the reason of supporting Kotlin's co-routines.

David Leopoldseder (on Oct 2022): … The optimizer is written based on our structured loop representation in loop closed SSA form, really supporting only reducible CF. … We do a limited form of support for detecting some common irreducible loops during PE and rewrite them to a switch pattern…

The quotes above explains several things in the IGV graphs of the static backtracking implementation:

• There are several huge IntegerSwitch nodes.
• The graph contains many Loop nodes.
• The whole graph is just super complicated.

And I guess it also explains the performance. Anyway, I guess we will need to base our engine on boolean values and proper loops then. (The implementation is in the git logs in case any one is interested.)

• pcomplete.el

  "\\(?:.\\|\n\\)*?\\(\\(?:.\\|\n\\)*\\)\x3FFF7F\\(?:.\\|\n\\)*\\1\\'"
  

• dirtrack.el

  "\\`\\(.*\\)\\(?:/.*\\)?\n\\(.*/\\)\\1\\(?:/.*\\)?\\'"
  

• diff-mode.el

  (concat
   "\\`\\(.*?\\)\\(.*\\)\\(.*\\)\n"
   "\\1\\(.*\\)\\3\n"
   "\\(.*\\(\\2\\).*\\)\\'")
  

https://news.ycombinator.com/item?id=42207282

https://emacsninja.com/posts/code-conversion-language.html

• [X] Emacs Lisp reader (lexer & parser)
• [X]

  Buffer-local variables & scoping

  Basically, in addition to buffer-local variables, forwarded variables, etc., we want to add transparent "thread-local" variables, so that:

  • Dynamically bound variables are thread-local, during the lifetime of which other threads sees the original value.
  • Some specific variables must be thread-local to make transparent concurrentization work.
  • Also, lexical scopes are always thread-local.
  • [X] Handle default values
• [X] All special forms
• [ ] Bootstrap loadup.el
• [X] A fallback, feature-complete regex engine -> no
• [-] Emacs Lisp byte-code interpreter in Truffle
  • [X] ELispBytecodeFallbackNode: a fallback bytecode interpreter with not-bad JIT performance
  • [ ] A bytecode interpreter based on nativecomp?

Hopefully we don't need to take too much effort to implement these tons of types. We might need some boilerplate code for strings / integers for Truffle interop, but otherwise simply using some classes with public member should do.

• [X] Lisp_Symbol
  • Constant marker
  • Intern state
  • Special?
  • Name
  • Value (cache)
    • Types:
      • Plain var
      • Varalias
      • Localized var (buffer local variables)
      • Forwarding variable
  • Function value (cache)
  • Property list
• [X] Lisp_Int*
• [X] Lisp_String
• [-] Lisp_Vectorlike [13/36]
  • [X] PVEC_NORMAL_VECTOR
  • [ ] PVEC_FREE
  • [X] PVEC_BIGNUM
  • [X] PVEC_MARKER
  • [ ] PVEC_OVERLAY
  • [ ] PVEC_FINALIZER
  • [X] PVEC_SYMBOL_WITH_POS (maybe integrate into ELispSymbol)
  • [ ] PVEC_MISC_PTR
  • [ ] PVEC_USER_PTR
  • [ ] PVEC_PROCESS
  • [ ] PVEC_FRAME
  • [ ] PVEC_WINDOW
  • [X] PVEC_BOOL_VECTOR
  • [X] PVEC_BUFFER
  • [X] PVEC_HASH_TABLE
  • [X] PVEC_OBARRAY
  • [ ] PVEC_TERMINAL
  • [ ] PVEC_WINDOW_CONFIGURATION
  • [X] PVEC_SUBR
  • [ ] PVEC_XWIDGET
  • [ ] PVEC_XWIDGET_VIEW
  • [ ] PVEC_THREAD
  • [ ] PVEC_MUTEX
  • [ ] PVEC_CONDVAR
  • [ ] PVEC_MODULE_FUNCTION
  • [ ] PVEC_NATIVE_COMP_UNIT
  • [ ] PVEC_TS_PARSER
  • [ ] PVEC_TS_NODE
  • [ ] PVEC_TS_COMPILED_QUERY
  • [ ] PVEC_SQLITE
  • [X] PVEC_CLOSURE
  • [X] PVEC_CHAR_TABLE
  • [X] PVEC_SUB_CHAR_TABLE
  • [X] PVEC_RECORD
  • [ ] PVEC_FONT
  • [ ] PVEC_TAG_MAX
• [X] Lisp_Cons
• [X] Lisp_Float

In the past few months I've done quite some things to the codebase, including adding array cons and removing it, moving from mule-string to truffle strings, etc. Now, ELispLanguageTest has become very slow. I suspect that it is due to string-related commits, leading to increasing buffer load/parse time.

• Rationale #1: Personally I would like the renderer to be in Rust (because of the vast number of UI libraries there). However, there is a fundamental problem here: most Rust libraries (including std ones) assume UTF-8 coding, while we currently use a compressed UTF-32-like coding, not to mention our code points can go up to #x3FFFFF.

  • Edit: I don't think most of the Rust libraries are that good though. We want out of box support for:

    • Platform-native font fallback
    • BIDI display
    • IME support
    • Glyph level manipulation
    • Good accessibility

    I don't think these requirements are nitpicking, but, I mean, not even GTK supports them all (see A 2025 Survey of Rust GUI Libraries (13 Apr 2025)). Just WTF.

  • Another problem is that we need to pass strings to Rust via UTF-8 (or Java via UTF-16), and there will be a huge performance penalty if we are to encode strings every single time we redisplay things.
• Rationale #2: Both Emacs buffers and strings have text properties attached to them, and when a string gets inserted into a buffer, so does its properties. So we definitely do not want two separate interval tree implementation for buffers and strings.
• Rationale #3: We have a tree for our ropes, and we want another tree for intervals and maybe one more for our marks. What if we merge them?

I have stopped writing tests for newly implemented subroutines. We should definitely try to load ert.el before continue to bootstrap loadup.el.

There is definitely a bug, but it is rather unreproducible. Can't deterministically trigger it. Have no idea what could be the reason (maybe something depending on hash table iteration order?).

(cl--generic-cyclic-definition cl-generic-generalizers)

Probably fixed: this seems related to wrong defvar semantics. I haven't seen the error ever since.

Now that Truffle has a bytecode DSL, we can probably implement a bytecode interpreter more easily. However, I don't think anything (Juicemacs currently can run) in Emacs actually requires a working bytecode interpreter. Since our interpreter is more or less on par with nativecomp when fully warmed up, we can continue with our AST interpreter until we run into some real bottlenecks.

Edit: I was benchmarking using floats, which Emacs is known to be bad at. In integer-based tests our interpreter is significantly slower than nativecomp.

Edit: No, bytecomp.el requires a working bytecode interpreter.

Edit: No, I misunderstood how Emacs bootstraps so the following sub-sections are speaking nonsense. It does dump twice, but the two dumps are rather independent:

1. pbootstrap:
   • temacs -> (load "loadup") -> (dump-emacs) -> bootstrap-emacs
   • bootstrap-emacs is used to byte-compile files into .elc
2. pdump:
   • pdump expects .elc files, but otherwise it has nothing to do with bootstrap-emacs. (I wrongly thought bootstrap-emacs is used to dump a second time.)
   • pdump -> (load "loadup") (.elc) -> (dump-emacs) -> emacs

EDIT: This is cancelled. Let's simply try to align with the Emacs bytecode compiler: this is unsupported.

Currently in GNU Emacs 29, a function may be a lambda/closure cons:

;;; -*- lexical-binding: t -*-
(defalias 'my-func #'(lambda () 42))
(symbol-function 'my-func)

(lambda nil 42)

With lexical binding, you will get (closure (t) nil 42) instead of (lambda nil 42).

Since it is a cons, you may modify it to change the function definition on the fly. For the GNU Emacs interpreter, since it literally evaluates the cons, it is fine. But for Truffle interpreters, which usually wants a static AST for better JIT performance, this is bad news.

Currently, we don't bother checking whether each AST node needs update and the function may be considered immutable once the whole AST is constructed. (One exception is that we check at each cons node (maybe-function arg1 ...) whether the target function is changed so as to handle subroutines, functions as well as macros.) Also, with our interpreter, (function (lambda ())) produces <a closure object> instead of a cons, which is another behavioral difference.

The current performance is pretty good and we might consider adding more checks to ensure a consistent behavior. But we will need some benchmark to ensure it does not slow down too much.

Current implementation of lexical scoping spends too much time book-keeping. For example, for (while ... (let ((x (fun))) ... )), we want it to be compiled to something like:

//while block start
////let block start
frame.setSlot(xSlot, resultOfFun); // xSlot: CompilationConstant
//...
////let block end
//while block end

That is, we want each let clause to be compiled to a single stack frame assignment instruction. However, our current implementation is compiled to:

//...
////let block start
ELispLexical lexical = ELispLexical.getLexical(frame);
int xSlot = lexical.addVariable(X_SYMBOL);
frame.setSlot(xSlot, resultOfFun);
////let block end

This is because we need to support per iteration scope in case of closure creation inside a loop:

(while (some-condition)
  (let ((i (some-value)))
    (push (lambda () i) closures)))

Note that for each different (lambda () i), i is a totally different variable. As far as I know, there are two ways to handle this:

1. Use the same stack frame, but use non-constant frame slot number for each i. This is what we do now.
2. Use constant frame slot for every variable, and copy the whole frame when needed (i.e., when there is lambda creation in a loop). This is what GraalJS does.

The latter approach seems significantly more complex. (Considering GraalJS can know if a node needsPerIterationScope at parse time, while we have to do this at runtime, it only gets worse.) But it is more performance in that it should compile normal let clauses (without closure creation) into a single instruction.

Also, currently our markers does not move when texts get inserted/deleted.

Sidenote: I think we do not need those fany CRDTs or OTs. It seems that we cal simply keep some special thread-local marks/properties to make edits in a threaded context invisible to other threads. So each thread enjoys their own buffer, while the UI can "redisplay" changes by all threads (or changes done prior to redisplay).

The C implementation of load is quite simplistic: similar to internal-make-interpreted-closure-function, Emacs relies on load-source-file-function to handle more complex scenarios and file encoding.

FileSystem is used by users. And I assume, as a language implementer, we should use the methonds in TruffleLanguage.Env instead.

So a previous commit (3465a76 perf: use assumptions for frame materialized top tracking) introduced a bug: let/let* statements should have N + 1 assumptions instead of only one, since the value branches can also modify the stack and introduce more variables.

This is not revealed until we implement this function storage caching.

(BTW, the cache brings current loadup.el execution from ~7s to ~4s. Hopefully we are not getting things seriously wrong here.)

• [X] Avoid several duplicate allocations, esp. new Object[] for function arguments (incomplete optimizations done)
• [X] (let/let* () ...) is equivalent to (progn ...).
• [X]

  Concurrent class loading

  Built-in function initialization loads thousands of classes. Since each factory is independent of each other, we can make them concurrent. (The init function used to take ~0.5s, and now it takes ~0.3s. Not much, but still good to have and fun to concurrentize things.)

Currently, we heavily use static variables and basically everything is global. This is the Emacs way: single threaded, globally dynamically bound. However, this has already posed a few challenges:

• Truffle assumes the language allows several parallel contexts, that is, we can execute i = 1; print(i) and i = 2; print(i) concurrently without them interfering with each other. Apparently, our "global state for everything" approach can be problematic.
  • Previously, before we auto-gen the giant mess of initialization code, since JUnit tests are not concurrent, we are mostly fine, as long as we clean things up when creating a new context.
  • Now that we have convoluted init logic, with global variables scattered around the place, it becomes harder to properly "clean things up". (The tests are now failing with tons of (fatal) errors, probably due to charset initialization.)
• Although we plan to follow JavaScript's model of concurrency: single-OS-threaded green threads, it will be nice to have real thread APIs, similar to the Web Worker API. Then, it will be necessary to separate the dynamic variable scope of different "workers".

See emacs-extractor.

For string processing in any language, I suppose there are always at least two kinds of offsets: char offsets & code point offsets. Emacs basically uses byte offsets and codepoint offsets, but Java uses UTF-16 char offsets and codepoint offsets. So to correctly handle strings in Juicemacs, we need to incorporate all these three kinds of offsets.

It could be easier if we could just follow Emacs. (And Truffle actually provides a byte-offset based TruffleString to free us of the trouble!) But, unfortunately, we can't. There is one indispensible Java API we need to use: Java Pattern regex, which, of course, is based on UTF-16 char offsets.

Edit: In order to support the widened code point range in Emacs, we are now rolling out our own string implementation. The good news is that, many Emacs string operations actually involves case tables and all, requiring a reimplementation of most of the java.lang.String API anyway. So why not?

Oh no. It looks like a data structure with its internal structure fully exposed as a vector. It probably means we will have to exactly follow the GNU Emacs implementation.

Edit: No, it is not fully exposed. (See keymaps for what is fully exposed… Sigh.) It is a hash-table-ish thing. And actually, I do think we can make all hashes zeros and use our own HashMap instead.

It seems basically a char table. However, in order to use it with RegExp, we will need to maintain a character set for each syntax class, which might take quite some memory if unoptimized.

Edit: task cancelled now that we have a dedicated regex engine.

• Maybe set ELispContext.NIL to Boolean.FALSE (and T to Boolean.TRUE).
• [X] elisp/scripts/extract-emacs-src.py
  • Auto-detect types
  • Avoid java keywords
  • Auto-update existing ones
  • Set generated return types to Void
• Analyze null values during initialization

A central mechanism to handle exceptions.

I guess I should start implementing it before I litter IllegalArgumentException everywhere.

• [X] New exception types & utility methods
• [X] signal/error/condition-case
  • [X] Implementation
  • [X] Error groups
  • [X] Convert ClassCastException to errors
• [X] catch/throw
• [X] Stack traces
  • [X] Store debug info into cons nodes.
  • [X]

    So we want function names in stack traces, but all interpreted functions are just lambdas in Emacs: (defalias 'a-symbol #'(lambda () ...)). Maybe we can try to assign a lambda a name when it is first bound to a symbol? (<- chose this approach)

    I will need to check out how GraalJs handles lambdas. Too lazy to do that.

• Threefold speed-up: (fib 35) went from ~5s to 1.7s. At least we are not slower than interpreted GNU Emacs now (~3s).
  • Any other languages I tested takes less than an instant.
  • Python 3.12 takes ~0.6s. So it is quite embarrassing that a JIT implementation cannot beat an interpreted language.
    • JMH results: ~0.4 s/op, probably jacoco is interfering with previous results.
    • But… (mandelbrot 750) takes around 5.5s while Python uses only a second. So there is definitely space for improvement. (Emacs: ~30s.)
• Writing a Language in Truffle. Part 3: Making my Language (Much) Faster

I really doubt I get the implementation of ELispSymbol wrong (to some degree). Basically, a symbol can:

• Contain a lisp value (plain value symbol)
• Point to a field in a global C struct (forward symbol)
• Point to a field in a buffer struct (buffer-local symbol)
• Contain a user-defined buffer-local symbol (buffer-local symbol)
• Point to another symbol (aliased symbol)

Also, similar to Java, lisp functions and values are in different "namespaces". So in the function namespace, a symbol can:

• Point to a C function
• Point to a lisp function
• Point to another symbol (aliased function)
• Other special values:
  • Macros
  • Autoload functions
  • Wait, what? A keymap?
• Other values set by defalias

I have no idea how all these things interacts. (For example, what happens when you try to set the buffer-local default value for a plain value symbol? What if it is lexically bound? What behaviors may change if a symbol is lexically bound?)

(Did I forget to mention that symbols like :keyword are automatically constant? Or are they? They also seems to evaluate to themselves.)