lumen
A dynamically-typed scripting language in Kotlin. Source is scanned, parsed into an AST, compiled to bytecode and run on a stack-based virtual machine.
A dynamically-typed scripting language in Kotlin that genuinely runs code: scanning and parsing to an AST, compilation to bytecode and execution on its own stack-based virtual machine.
Overview
lumen is a dynamically-typed scripting language written in Kotlin that genuinely runs code. Source is scanned, parsed into an AST, compiled to bytecode, and finally run on its own stack-based virtual machine. It looks like an ordinary scripting language, but underneath every line goes the full way from text to machine instructions.
The difference between lumen and a typical language-writing exercise lies in one word: bytecode. It is easy to build an interpreter that walks a tree and evaluates it on the fly. lumen goes a step further and compiles, which is both harder and far more interesting.
A tree-walking interpreter is simple to write, but quickly hits a performance ceiling. Every run of the same loop means walking the same tree nodes again, with all the overhead that carries. For a small script it is invisible, for anything larger it is felt.
lumen takes the harder but better road: instead of executing the tree, it compiles it to bytecode, a flat list of simple instructions. Those instructions then run on a virtual machine that no longer has to understand any syntax - it gets ready-made commands and simply executes them.
From text to bytecode
The whole path breaks into four stages. The scanner cuts text into tokens, the parser arranges them into a tree, the compiler turns the tree into bytecode, and the virtual machine executes that bytecode instruction by instruction. It looks like ordinary code, but every line goes the same full way.
source text becomes tokens.
tokens arrange into a syntax tree (AST).
the tree turns into bytecode.
the bytecode runs on a stack, instruction by instruction.
The stack machine from the inside
A stack machine has no registers like a real CPU - it pushes and pops operands on a stack. Addition comes down to three moves: push two numbers, pop them and add, push the result. The same model powers Python and Java underneath, among others, so writing one yourself is a look into how the languages you use every day actually work.
Addition on a stack machine
| Step | Instruction | Stack after |
|---|---|---|
| 1 | push 2 | 2 |
| 2 | push 3 | 2, 3 |
| 3 | add | 5 |
A stack machine works without registers, and yet can execute any program. All the arithmetic and value passing happen through a single stack - surprisingly simple, and at the same time exactly the idea that powers large production languages.
The result: "compiled" stops being an abstraction
What comes out is a language that genuinely runs code - it does not interpret a tree on the fly but compiles it and runs it on its own machine. It is a project after which the word "compiled" stops being an abstraction, because you walked the whole way yourself, from a character in a file to an instruction that does something. A small language, but complete all the way down to execution.
More projects
More work from the same category - see how we tackle similar challenges.
Have a similar project?
Get in touch - a quote is free and comes back within an hour.