Skip to content

Tim 2: new grammar, checker, portable runtime and one code generator for six platforms - #3

Open
vmaxer wants to merge 23 commits into
mainfrom
claude/charming-thompson-4xd5p3
Open

Tim 2: new grammar, checker, portable runtime and one code generator for six platforms#3
vmaxer wants to merge 23 commits into
mainfrom
claude/charming-thompson-4xd5p3

Conversation

@vmaxer

@vmaxer vmaxer commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Implements the top-10 list, built bottom up: grammar, then lexer and parser, checker, runtime, code generator, platforms, CI and docs.

Language (GRAMMAR.md)

  • Tim 2 grammar. Precedence runs from |> down to **, and comparisons chain. The operators are C-style bitwise & | ^ ~ << >> (arithmetic shift), ** for power, f(x) = body definitions, and matching with => arms, ~> defaults and | guard => lines. Loops use @, with break/continue @N and optional ! N bounds.
  • Booleans are the numbers 1 and 0. Strings are UTF-8 byte strings. There is one list layout and one map layout.
  • examples/ and the test programs are migrated to the new syntax.

Front end

  • lexer.go, parser.go: rewritten. The lexer tokenizes up front and decides where statements end. The parser is recursive descent with positions, statement-level error recovery and precise messages.
  • check.go (new): checks name resolution, closure captures (mutable captured variables are boxed), immutability, same-block redefinition, arities and definite type errors, with "did you mean" suggestions. It also rejects x = x + 1 in a loop body, which would silently shadow an outer x.

Runtime (runtime/rt.c)

  • A freestanding C library with no libc and no writable data. It provides NaN-boxed values, exact integers and rationals, portable math, UTF-8 strings, lists and maps, errors with messages (error: cannot add a num and a str), printf, and the builtin library.
  • Floats print as the shortest decimal that reads back exactly (matched Python on 75k random doubles).
  • A non-moving mark-sweep garbage collector: conservative stack scan, a start bitmap for interior pointers, precise tracing inside objects, and size-class free lists. 3M allocating iterations stay at 66 MB.
  • runtime/build.sh builds relocation-free blobs, plus Go symbol tables, for each OS and architecture.

Code generator

  • core.go sits over a small asm interface. Thin emitters implement it: core_x86.go, core_a64.go, core_rv64.go.
  • It generates closures, tail calls that reuse the incoming argument area, and inline fast paths for plain-double arithmetic and comparisons. A prelude written in Tim provides map, filter, fold, any, all and sort with a key.
  • Speed is on par with the legacy backend: fib(30) 23 ms vs 25 ms, a 30M loop 0.21 s vs 0.24 s.
  • Output formats: static ELF for Linux x86-64/arm64/riscv64, PE for Windows x86-64/arm64 (kernel32 through the IAT, ASLR-capable), and Mach-O for macOS arm64 (libSystem through chained fixups, ad-hoc signed).
  • The driver uses the core whenever the checker accepts a program without legacy features. Programs that use C FFI, cstructs, unsafe or defer fall back to the legacy backends. TIM_LEGACY=1 forces legacy.

Platforms, tests, docs

  • .github/workflows/ci.yml: Linux runs go test ./..., with arm64/riscv64 under qemu. A separate job tests the runtime natively and rebuilds the blobs. The core programs also run natively on macOS 14, Windows x86-64 and Windows arm64.
  • testdata/core/*.tim programs, with hand-checked .want outputs.
  • README is rewritten. GRAMMAR.md is the reference. The new STDLIB.md lists every builtin, and a test checks it against the builtin table. LANGUAGESPEC.md and FUTURE.md (Tim 1) are removed.
  • About 570 unreachable legacy functions and several dead AST nodes and files are deleted.

Behaviour changes

  • Tests encoding Tim 1 behaviour are updated for these:
    • pop(xs) returns the element.
    • 0 / 0 is an error.
    • Out-of-range indexing is an error.
    • Maps print their keys.
    • >> is arithmetic.
    • %g prints the shortest form.
  • A binding's value is the bound value, so x = 42 as the last statement exits with 42.

Testing

  • go test ./... passes locally, including cross-arch under qemu.
  • CI is green on all jobs: Linux, runtime, macos-14, windows-2022, windows-11-arm.
  • A differential sweep of examples/ shows the core matching legacy output on every program it handles.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp

Generated by Claude Code

claude added 23 commits September 25, 2026 07:56
@claude
Tim 2 grammar, lexer and parser
344ae3e 
GRAMMAR.md is rewritten as a small, precise grammar that lexer.go and
parser.go implement exactly. The lexer tokenizes the whole file first and
decides which newlines end statements (inside brackets, after an
operator, before a line starting with |>, ., or!, and, or they do not);
the parser is a recursive-descent parser over that token list with
statement-level error recovery and positioned, specific messages
("expected an expression after '+'"). It produces the existing AST.

Language changes (GRAMMAR.md section 8): C bitwise operators replace the
b-suffixed ones; |> is the pipe; bit/rotl/rotr/random() replace ?b,
<<<b, >>>b and ??; f(x) = body definitions and Type.method(self) = body
methods replace fun; comparisons chain; `not in`; negative literals;
list comprehensions; nested block comments; digit separators and 0o;
field: type in cstructs and parameters; break/continue @n replace ret @n;
~> is the only default arm; loop bounds are optional everywhere; guard
lines work in any block. Removed: ^ as power, ::, <>, ||, ENet
literals and send/receive, ¤, µ, $, ??, x++, postfix #, call()!, call
recursion bounds, class, with, alias, spawn, shadow, foreach/for/while,
@first/@last/@counter/@i and the : cast shorthand.

Fixes found on the way: string escapes were processed twice (a literal
\\n became a newline); `@ { }` stopped after 1,000,000 iterations; a
match subject was re-evaluated per arm (now evaluated once when it has
side effects); cast types were limited to a fixed list.

Tests and examples are migrated to the new syntax; tests for removed
features are deleted. Unreachable code (571 functions, 22 files and the
unused internal/engine package) is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add the semantic checker and the standard library's builtin table
751ec9d 
check.go resolves every name to a global, local, parameter or closure
capture (boxing mutable variables that closures capture), enforces
immutability and single definition per block, checks call arities of
known functions and builtins, infers the static types it can, and reports
definite type errors with positions and "did you mean" suggestions. It
also records features the new code generator does not handle yet, so the
driver can fall back to the legacy backends.

AST nodes that diagnostics refer to now carry source positions, and
ParseProgramRaw returns the AST before the legacy optimizations.
builtins.go lists the Tim 2 standard library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add the portable Tim 2 runtime
d1bd9c5 
runtime/rt.c holds every operation on Tim values that is not a machine
instruction: NaN-boxed values, exact integers and rationals, portable math,
UTF-8 strings, one list and one map layout, errors with messages, shortest
round-trip float printing, printf and the builtin library. build.sh turns it
into relocation-free blobs for x86_64, arm64 and riscv64 with a table of
exported functions; test/ runs it natively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add the core code generator and the x86-64 emitter
3910343 
core.go compiles checked Tim 2 programs through a small asm interface: NaN-boxed
values with inline fast paths for plain doubles, closures with boxed captures,
tail calls that reuse the incoming argument area, and every other operation in
the portable runtime. core_x86.go is the x86-64 emitter, core_elf.go writes a
static executable with no writable segment, and the Tim prelude supplies map,
filter, fold, any, all and sort with a key.

The driver uses the core when the checker accepts a program with no legacy
features and falls back to the legacy backends otherwise. The checker now
rejects 'x = x + 1' in a loop body that would shadow an outer x, and tests
follow Tim 2 semantics: pop returns the element, 0 / 0 is an error, floats
print as the shortest round-trip decimal and maps print their keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add arm64 and riscv64 emitters for the core code generator
fe3dffb 
Both are thin: registers, loads and stores, integer and float operations,
branches and calls, with label fixups for each branch form. Every program the
core accepts now compiles for x86-64, arm64 and riscv64 from one code
generator, and the cross-architecture tests agree on Tim 2 semantics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add a garbage collector to the runtime
8c19bde 
A non-moving mark-and-sweep collector: the heap is a list of chunks with a
bitmap of object starts, so a conservative scan of the machine stack can find
objects from interior pointers, while fields inside objects are traced
precisely by kind. Sweeping coalesces dead objects into size-class free lists.
Collection runs when the words allocated since the last one exceed the live
size (at least 32 MiB), so the heap stays within about twice the live data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add Windows and macOS arm64 to the core, a core test corpus and CI
6b9b51a 
Windows x86-64 and arm64 executables call kernel32 and advapi32 through an
import address table, and macOS arm64 executables call libSystem through a
global offset table bound by chained fixups and are ad-hoc signed. The runtime's
OS functions take the OS itself, so start code can reach the imports without
writable data in the blob. testdata/core holds programs with their expected
output, run natively and under qemu, and a GitHub Actions workflow runs the
test suite on Linux and the core programs on macOS and Windows. The >> operator
is now an arithmetic shift, and '@ ! N { }' parses as the grammar says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Trim the documentation to Tim 2
f2e1b2a 
README.md is a short, accurate introduction; GRAMMAR.md is the language
reference and STDLIB.md lists the builtins, checked against the builtin table
by a test. LANGUAGESPEC.md and FUTURE.md described Tim 1 and are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Commit the expected outputs of the core programs as .want files
d70204d 
*.out is ignored by .gitignore, so the golden files never reached CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Keep testdata/core byte-exact on Windows checkouts
5bf85bc 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Make core PE images relocatable for Windows on ARM64
fb4ac64 
ARM64 Windows refuses images that do not allow ASLR. The code is position
independent, so executables now carry a relocation table with one empty
block, set dynamic base and high-entropy VA, and target Windows 8 (6.2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add raw strings and sharper diagnostics
0abecb8 
Backquoted raw strings have no escapes and may span lines. Undefined names
are reported by the checker, with its positions, unless sibling files could
define them; names from other languages (len, append, true, nil) get a hint
at the Tim way, and suggestions allow about one edit per three characters so
short names do not attract unrelated ones. Programs that use legacy features
go to the legacy backends even when the checker also sees errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Make % floored, 2 ** 0.5 exact to the ulp, and num reject '.'
1a0d6d3 
The remainder now takes the divisor's sign, as in mathematics, Python and
Haskell: -7 % 3 is 2. A power of one half is a correctly rounded square
root, and num requires at least one digit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Compile defer in the core; exit with 0 unless asked otherwise
db68d0f 
defer registers a function of no arguments that runs when the enclosing
function returns, or when the program ends, last first; functions with
deferred calls do not make tail calls. The exit code no longer comes from
the last statement, which made a script ending in 'n <- n + 1' fail: it is
0 unless the program calls exit, returns at top level or main gives one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Call C from the core: dynamic imports, rt_ffi and memory builtins
7a5f964 
C functions are called through rt_ffi, which converts each argument by its
C type and places it for the platform's calling convention by calling
through a function pointer declared with integer, float and stack slots
(sixteen positional shapes on Windows x64). Signatures come from headers or
a table of C library functions; without one, integers pass as integers and
other numbers as doubles. Executables that call C import the functions:
ELF gains an interpreter, DT_NEEDED and GLOB_DAT relocations, PE more DLLs,
and Mach-O more dylibs in its chained fixups. C constants resolve at compile
time, ptr values support + and -, and read_*, write_* and cstr access memory.
C's stdio is flushed before Tim's output, so the two stay in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Compile cstructs in the core
c61cce6 
The checker tracks which cstruct a value points to: from constructors, as,
parameter annotations, self, typed loop variables, lists of cstructs, nested
fields and functions that always return one. Field reads and writes compile
to typed memory accesses at their offsets, constructors allocate on the
garbage-collected heap, methods resolve on typed receivers, and +, -, * and /
on cstructs call their add, sub, mul and scale methods. Indexing a cstruct or
reading a field of an unknown one is a compile error; examples and tests
that relied on positional fields now name them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Import msvcrt's names for C99 functions; install the riscv64 C librar…
32a9413 
…y in CI

msvcrt.dll exports snprintf, hypot and friends with a leading underscore;
a program importing the C99 names failed to start with a missing entry
point. The Linux job needs the riscv64 C library to run dynamically linked
riscv64 programs under qemu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Compile the remaining examples in the core
4daf143 
alloc(n) gives zeroed bytes on the garbage-collected heap, exitf prints to
stderr and exits with 1, casts to float, double, cptr and cstring work, and
casting a ptr to an integer gives its address. A program may span the .tim
files of its directory: definitions of undefined names, with their files'
cstructs and C imports, are taken from sibling files. C type annotations are
accepted, and the examples use read_u32 and c.malloc instead of Tim 1 names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Add popcount, clz and ctz; move the last tests off Tim 1 builtins
3ee09e4 
popcount, clz and ctz count bits of 64-bit integers. Memory builtins accept
an address given as a number, and the C library table knows mmap, fork,
waitpid and _exit. Tests use Tim 2 spellings (xs + [v], xs[1:], exitf,
c.malloc, byte offsets), so the whole test suite now compiles with the core
code generator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Compare core program output without C's Windows line endings
fc14e21 
msvcrt's text-mode stdout writes \r\n for C's output; Tim writes \n.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Compile only with the core; Tim modules in the core; remove unsafe an…
4bad2ee 
…d vec2/vec4

CompileTim now uses the core code generator alone. Tim imports resolve to a
library, git repository or directory whose definitions become alias.name
(unprefixed for 'as *' or a module that says 'export *'), with references
inside the module renamed to match. 'unsafe' register blocks and vec2/vec4
are removed from the language: C functions and read_*/write_* cover them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Delete the legacy backends; real arena regions; a small CLI
3f5dad3 
The core is the only compiler, so the legacy code generators, instruction
encoders, object writers, optimizer and their tests are gone (about 75k
lines), with the vendored x/sys and the numrt blob.

arena { } is now a real region allocator instead of a no-op. Allocation
inside is a pointer bump into region chunks; a write barrier records older
objects that get a region pointer; leaving the arena traces only from the
stack, the globals, those objects and the arena's value, returns chunks
without survivors to a pool and promotes the rest to the enclosing arena or
the heap. Long arenas collect themselves. Per-frame garbage in an arena runs
2x faster in a sixth of the memory of the plain collector.

The CLI is build/run/test/-c with flags anywhere, --target, --watch that
reruns on change, #! scripts, and diagnostics colored only on terminals.
Removed Tim 1 builtins report what to use instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
@claude
Direct C calls, inline syscalls, platform()
bd62175 
A C function with a known signature whose arguments fit in registers is now
called directly through its import entry: arguments convert inline (plain
doubles to integers, ptr untagging, floats as they are) and so do results,
with the runtime only for big exact numbers, strings and NaNs. Ten million
libc calls take 0.09s instead of 0.51s. Variadic and other calls keep
rt_ffi. long is 32 bits on Windows.

syscall(n, ...) is a Linux system call in one instruction, and platform()
names the target, so a program can pick its syscall numbers.

The asm interface gains conversions, floating-point argument and result
moves, arithmetic shifts and syscall on x86-64, arm64 and riscv64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q77FvuYDeiyYVBozLKTBpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vmaxer @claude