Code tour: from `.peep` to an executable

A guided walk through the Peeper compiler, following one source file all the way to a native binary. Read it top to bottom the first time; after that, jump to the phase you need.

Every code sample here is simplified — real signatures carry more parameters and more error handling. Each one names the file it came from so you can read the real thing.

Related reading: RULES.md for mandatory engineering requirements, COMPILER_GUIDELINES.md for design-review guidance, docs/compiler-architecture.md for current architecture, and docs/compiler-framework/change-paths.md for a file-by-file change guide. Verify mutable details against source.

1. The thirty-second version

flowchart LR
    SRC[".peep source"] --> LEX[lexer]
    LEX --> PAR[parser]
    PAR --> AST[AST]
    AST --> SEM[semantic analysis]
    SEM --> THIR[THIR]
    THIR --> CFG[control-flow graph]
    CFG --> ANA[flow facts · effects · ownership]
    ANA --> MIR[MIR]
    MIR --> LL["LLVM IR text"]
    LL --> CLANG["clang"]
    CLANG --> OBJ["object files"]
    OBJ --> LINK["linker"]
    LINK --> EXE["executable"]

The compiler itself stops at LLVM IR text. Turning that into a binary is clang, invoked as an external tool. Peeper does not link anything by hand.

2. Package map

Package	What lives there
`cmd/`	CLI commands: `build`, `run`, `check`, `dump`, `doctor`
`internal/driver`	Thin wrapper that builds a `CompilerContext` and compiles one file
`internal/pipeline`	Module loading and the phase ladder that drives everything
`internal/module`	`Module` and its phase artifacts
`internal/project`	`CompilerContext`, module registry, imports, and fingerprints
`internal/frontend`	`token`, `lexer`, `parser`, `ast`
`internal/semantics`	Collector, binder, resolver, const eval, typechecker, flow facts, effects, definite init, ownership, usage, plus the artifact packages
`internal/ir`	`thir`, `cfg`, `exprlower`, `mir`, and the shared `ir` node/type model
`internal/backend/llvm`	MIR → LLVM IR text
`internal/toolchain`	Finds `clang` and the sysroot; builds its command lines
`internal/diagnostics`	Errors, warnings, source rendering, phase attribution
`runtime/`	`peeper_rt.c` — the C runtime linked into every binary
`_builtin_library/core`	The `core` standard library, written in Peeper

3. Entry: command → driver → pipeline

sequenceDiagram
    participant U as user
    participant C as cmd/build.go
    participant D as driver
    participant P as pipeline
    participant T as toolchain

    U->>C: peeper build main.peep
    C->>D: CompileFile(ctx, path)
    D->>P: pipeline.Run(ctx, entry)
    P-->>D: every module at phase.Backend
    D-->>C: modules carrying LLVMIR
    C->>T: Resolve(clang, sysroot)
    C->>T: clang -c mod_0.ll -o mod_0.o
    C->>T: clang @objects.rsp -o app
    C-->>U: executable

driver.CompileFile is deliberately small — it exists so the CLI, the LSP and the tests all enter the compiler the same way.

4. Module loading

Before any phase runs, the loader walks imports and builds a dependency graph.

// internal/pipeline/loader.go (simplified)
func (l *moduleLoader) Load(entry *module.Module) error {
    l.enqueue(entry)
    for len(l.queue) > 0 {
        module := l.pop()
        l.loadModule(module)      // read the file, lex, parse
        l.resolveImports(module)  // each import becomes a graph edge + a queued module
    }
    return nil
}

Modules are identified by moduleid.ID — origin, namespace, dependency and import path — never by file path, so a module keeps its identity if the project moves on disk.

The dependency graph is the shared internal/graph package. pipeline.Run topologically sorts it so a module is only advanced once everything it imports has reached the phase it needs.

flowchart TD
    E["entry: main.peep"] --> A["import core/io"]
    E --> B["import ./util"]
    B --> C["import core/mem"]
    P["prelude"]:::pre
    E -.always depends on.-> P
    A -.-> P
    B -.-> P
    classDef pre fill:#eef,stroke:#88a

Every module is given an edge to the prelude, which is how the prelude is guaranteed to be first in topological order without any special case in the scheduler.

5. The phase ladder

This is the spine of the compiler. Each module carries a Phase, and advanceModulePhase moves it forward exactly one step per call.

flowchart TD
    Setup --> Load --> Parsed --> Collected --> Bound --> Resolved
    Resolved --> Typechecked --> CFG --> Analyzed
    Analyzed --> Usage
    Usage --> MIR --> Backend --> Finalize

// internal/pipeline/pipeline.go (heavily simplified)
func advanceModulePhase(ctx *project.CompilerContext, module *module.Module) bool {
    if module.Phase < phase.Collected { collector.Collect(ctx, module); module.Phase = phase.Collected; return true }
    if module.Phase < phase.Bound     { binder.Bind(ctx, module);       module.Phase = phase.Bound;     return true }
    if module.Phase < phase.Resolved  { resolver.Resolve(ctx, module);  module.Phase = phase.Resolved;  return true }
    // ... one block per phase, in order ...
}

Why one phase per call? Because modules advance in lockstep across the whole project. A module cannot be typechecked until every module it imports is typechecked, and the scheduler enforces that by advancing everyone one rung at a time.

What each phase produces

Phase	Produces	Stored on `Module` as
`Parsed`	syntax tree	`AST`
`Collected`	top-level symbols, method sets	`SymbolIndex`, `ModuleScope`
`Bound`	operator/interface bindings	`SymbolIndex`
`Resolved`	every identifier → symbol	`SymbolIndex` occurrence lookup
`Typechecked`	types, typing decisions, finalized constants, typed source IR	`SymbolIndex`, `THIR`, `SemanticExportFingerprint`
`CFG`	blocks, sites, edges	`CFG`
`Analyzed`	flow, effect, definite-init, and ownership evidence	`Analysis`
`Usage`	usage diagnostics at the project barrier	diagnostics; no durable module artifact

Phase artifacts are the central design idea. Each durable artifact is produced by one owning phase and read by later phases. Flow, effects, definite initialization, and ownership evidence are published together in Analysis; Usage is a project barrier for diagnostics rather than another durable module artifact.

Module.ResetToPhase clears downstream artifacts in phase order, so incremental rebuilds cannot leave stale evidence behind:

// internal/module/module.go (simplified)
func (m *Module) ResetToPhase(retained phase.Phase) {
    m.Phase = retained
    if retained <= phase.Parsed { m.ModuleScope = nil; m.SymbolIndex = nil }
    if retained < phase.Collected { m.typeDeclarations = nil }
    if retained < phase.Typechecked { m.THIR = nil; m.SemanticExportFingerprint = "" }
    if retained < phase.CFG { m.CFG = nil }
    if retained < phase.Analyzed { m.Analysis = nil }
    if retained < phase.MIR { m.MIR = nil }
    if retained < phase.Backend { m.LLVMIR = "" }
}

6. Front end: text → AST

// internal/frontend — the whole front end, in three lines
tokens := lexer.New(path, source, diag).Tokenize()
module := parser.New(path, tokens, diag).ParseModule()

The parser is recursive descent and error-recovering: on a syntax error it emits a diagnostic and produces a BadStmt / BadExpr node rather than bailing out. Later phases treat recovery nodes as inert, which is why the LSP can still offer completion in a file that does not parse cleanly.

AST nodes carry:

a source.NodeID — provisional after parsing, then function-owned and stable for callable syntax before semantic collection;
a Location — for diagnostics;
forEachChild — the one canonical child walk.

// internal/frontend/ast (simplified)
type ForStmt struct {
    Index, Value *Ident
    Iterable     Expr
    Cond         Expr
    Body         *BlockStmt
}

func (s *ForStmt) forEachChild(visit func(Node)) {
    visit(s.Index); visit(s.Value); visit(s.Iterable); visit(s.Cond); visit(s.Body)
}

Forgetting a field in forEachChild makes it invisible to ast.Inspect, so traversal behavior is covered by ordinary AST tests and source fixtures rather than a second parser for the compiler implementation.

7. Semantic analysis

flowchart LR
    C["collector<br/>declare top-level names"] --> B["binder<br/>operators, interfaces"]
    B --> R["resolver<br/>ident → symbol, scopes"]
    R --> K["const eval<br/>compile-time values"]
    K --> T["typechecker<br/>types + decisions"]

Collector walks top-level declarations and puts them in ModuleScope. Binder resolves declaration types, publishes method sets, and wires up operator/interface members. Resolver creates block scopes and maps every referencing identifier to its symbol:

// internal/semantics/resolver (simplified)
module.SymbolIndex.Bind(ident, symbol)
module.SymbolIndex.SetScope(block, scope)

Identity rule. SymbolIndex records resolved syntax occurrences, including declaration names and assignment targets. Lexical scopes remain responsible for name lookup, shadowing, visibility, and declaration order; downstream phases use the published node identity instead of rescanning symbols by AST pointer.

The typechecker does more than check types — it publishes decisions later phases depend on, so nothing has to re-derive them:

// internal/semantics/typechecker/evidence.go (private during Check)
c.evidence.RecordExprType(expr.ID(), typ)
c.evidence.RecordImplicitConversion(expr.ID(), conversion)
c.evidence.RecordForIteration(loop.ID(), iteration)

// internal/semantics/typechecker/thir_build.go materializes those facts on THIR;
// later phases query THIR instead of re-deriving source meaning.

8. Control-flow graph

CFG turns structured syntax into blocks, sites and typed edges.

flowchart TD
    subgraph entry [b0]
      S0["site 0: let x = 1"]
      S1["site 1: terminator (if x > 0)"]
    end
    entry -->|EdgeTrue| then["b1: then"]
    entry -->|EdgeFalse| els["b2: else"]
    then -->|EdgeNormal| join["b3: join"]
    els -->|EdgeNormal| join

A site is one ordered program point inside a block:

// internal/ir/cfg/model.go (simplified)
type SiteID struct{ Block, Index int }   // dense and positional

type Site struct {
    ID       SiteID
    Kind     SiteKind          // statement | scope exit | terminator | join
    NodeID   source.NodeID     // the AST node this point stands for
    ScopeID  source.NodeID
    Successors, Predecessors []Edge
}

Edges keep their meaning — EdgeTrue, EdgeFalse, EdgeVariantCase with a case index — so consumers never guess control flow from adjacency order.

CFG does not import the typechecker. It asks for the two facts it needs through narrow function types, which the typechecker result happens to satisfy:

// internal/ir/cfg/build.go
type BuildQueries struct {
    MatchCases          func(source.NodeID) ([]int, bool)
    LoopGuaranteedEntry func(source.NodeID) bool
}

cfg.Module.Validate() then checks the topology it produced — block identity, termination, adjacency in both directions, reachability — and raises an internal-compiler-error diagnostic rather than a user-facing one, because malformed topology is a compiler bug.

9. Effects: the semantic operation stream

This is the newest layer, and the one that makes later analyses construct-agnostic.

The problem it solves: definite initialization and ownership each used to walk the AST themselves, re-deriving what every construct did to a binding. Two walks that had to agree — and sometimes did not.

One producer now translates each CFG site into ordered operations:

// internal/semantics/analysis/effect_ops.go (simplified)
type Op interface{ effectOp() }   // sealed set

type Place struct {
    Root        *symbols.Symbol           // the binding …
    Temporary   source.NodeID             // … or the expression, for a value owning nothing
    Projections []place.OriginProjection  // .field, [index]
}

type Define  struct{ Symbol *symbols.Symbol; Initialized, OnEntry bool }
type Write   struct{ Place Place }
type Use     struct{ Place Place; Kind typeinfo.UseKind }
type Borrow  struct{ Place Place; Mutable, Argument, Raw bool }
type Discard struct{ Place Place }
type CallBegin struct{ Node source.NodeID }
type CallEnd   struct{ Node source.NodeID }

The producer is the only code that reads syntax to decide meaning:

// internal/semantics/analysis/effects.go (simplified)
func (b *builder) value(site cfg.SiteID, expr ast.Expr, kind typeinfo.UseKind) {
    switch node := expr.(type) {
    case *ast.Ident:
        if sym := b.queries.Symbols[node.ID()]; sym != nil {
            b.emit(site, Use{Place: Place{Root: sym}, Kind: kind})
        }
    case *ast.BinaryExpr:
        if b.queries.StringConcatenation(node.ID()) {    // decided by the typechecker
            b.value(site, node.Left, typeinfo.UseMove)   // concat consumes its left
            b.value(site, node.Right, typeinfo.UseRead)
            return
        }
        b.value(site, node.Left, typeinfo.UseRead)
        b.value(site, node.Right, typeinfo.UseRead)
    case *ast.CallExpr:
        b.emit(site, CallBegin{Node: node.ID()})
        b.value(site, node.Callee, typeinfo.UseRead)
        for _, arg := range b.queries.CallArguments(node) {
            b.argument(site, arg)                        // borrows if the parameter is a reference
        }
        b.emit(site, CallEnd{Node: node.ID()})
    // … one case per expression kind, then: default: panic(…)
    }
}

Three subtleties that are easy to get wrong, all learned the hard way:

Define.OnEntry — a parameter, or a match payload binding, exists before its site runs. It is created by the edge into the site, not by the site. Liveness must not treat that as a definition within the site or it ends a borrow one step early.
CallBegin/CallEnd — a call is a lifetime, not a position. A temporary created while computing an argument lives until the call completes. A flat list of uses has nowhere to hang that.
Place.Temporary — f().field projects out of a value that lives in no binding. Ownership treats that differently, so the vocabulary has to be able to say it.

10. The analyses that consume it

flowchart LR
    E["Effects"] --> DI["definite init<br/><i>is it initialized?</i>"]
    E --> OW["ownership<br/><i>moves, borrows, drops</i>"]
    CFG["CFG"] --> DI
    CFG --> OW
    OW --> CP["CleanupPlan"]

They currently share evidence, not solver machinery. Each has distinct lattice, join direction, and diagnostics. A shared solver would need to reduce real duplication without hiding those differences.

Definite initialization is a must-analysis: a symbol is initialized only if it is initialized on every path, so the join is intersection.

// internal/semantics/analysis/definite_init.go (simplified)
func transfer(ops []effectOp, in initState) initState {
    out := copyInitState(in)
    visitor := &initializationVisitor{current: out, shouldApplyState: true}
    for _, op := range ops {
        visitEffect(op, visitor)
    }
    return out
}

It contains no AST switch at all; source identity comes from source.NodeID.

Ownership tracks moves, loans and liveness, then writes the drop plan:

// internal/semantics/analysis/ownership_effects.go (simplified)
func (a *analyzer) applyEffects(node *site, st ownershipState, loans *loanContext) {
    ops := a.effects[node.cfgSite.ID]
    visitor := &ownershipEffectVisitor{a: a, node: node, st: st, loans: loans}
    for _, op := range ops {
        visitEffect(op, visitor)
    }
}

The result is the CleanupPlan — the single source of drop obligations over source values:

// internal/semantics/analysis/cleanup.go
type cleanupPlan struct {
    AfterScope   map[cfg.SiteID][]symbols.SymbolID
    BeforeReturn map[source.NodeID][]symbols.SymbolID
    BeforeAssign map[source.NodeID]struct{}
    DiscardedValue map[source.NodeID]struct{}
    // …
}

Lowering reads this plan. It never decides a drop for itself.

11. Lowering: THIR + CFG → MIR

THIR is typed and structured. It carries symbols, conversions, places, ordered arguments, match arms, and iteration plans published by semantic analysis.

CFG owns execution topology. MIR lowering joins CFG sites to THIR nodes by source.NodeID, lowers expressions through internal/ir/exprlower, and reads ownership cleanup plans. It does not re-read AST or reconstruct control flow.

MIR is flat: basic blocks, instructions, terminators — close to what a backend wants.

// internal/ir/mir/model.go (simplified)
type Instr interface{ instrNode() }       // Assign Store Print Drop DynamicArrayOp Call InterfaceCall
type Terminator interface{ termNode() }   // Jump Branch SwitchVariant Ret

MIR instructions and terminators are sealed by unexported marker methods, so an instruction can never be used where a terminator belongs. MIR lowering walks CFG sites and consumes cleanup plans to place drops.

flowchart LR
    A["AST + evidence"] --> T["THIR<br/>structured, typed"]
    T --> M["MIR<br/>flat blocks + terminators"]
    M --> L["LLVM IR text"]

12. Backend and linking

// internal/backend/llvm/emitter.go (simplified)
func GenerateLLVMIR(mod *mir.Module, …) string {
    for _, fn := range mod.Funcs {
        for _, block := range fn.Blocks {
            for _, instr := range block.Instrs {
                switch typed := instr.(type) {
                case *mir.Assign: emitValue(lb, typed.Value)
                case *mir.Drop:   emitDrop(lb, typed)
                // …
                }
            }
            switch term := block.Term.(type) {
            case *mir.Jump:   lb.branch(target)
            case *mir.Branch: lb.condBranch(cond, then, els)
            case *mir.Ret:    lb.ret(value)
            }
        }
    }
    return b.String()
}

The emitter produces LLVM IR text, not bitcode. Then cmd/build.go shells out:

// cmd/build.go (simplified)
for i, module := range modules {
    os.WriteFile(fmt.Sprintf("mod_%d.ll", i), []byte(module.LLVMIR), 0o644)
    runCompilerTool(profile.ClangPath, profile.ObjectArgs(llPath, objectPath, debug))
}
profile.WriteResponseFile(responsePath, objectPaths)
runCompilerTool(profile.LinkerPath, profile.LinkArgs(responsePath, stagedPath))

internal/toolchain finds a managed clang and sysroot if one is installed, and falls back to clang on PATH with a warning. The C runtime in runtime/peeper_rt.c is linked in to provide allocation and printing.

Object files go through a response file rather than a long command line, which keeps the link working on platforms with tight argument limits.

13. Guardrails

The compiler is built so that forgetting something fails loudly.

Guard	Where	Catches
AST traversal tests	`internal/frontend/ast` tests + source fixtures	broken child traversal behavior
Sealed semantic type contract	Go type system	missing child/ownership behavior on a new semantic type
THIR validation + required operation methods	`internal/ir/thir`	malformed evidence or missing consumer support
MIR/backend rejecting dispatch	`internal/ir/mir`, `internal/backend/llvm`	unsupported lowered nodes fail loudly
`cfg.Validate`	`internal/ir/cfg`	malformed topology
`Analysis` validation	`internal/semantics/analysis`	malformed effects or cleanup evidence

A contract failure reads like this:

THIR operation contract has no implementation for a new node kind; add its case or
declare why the kind is inert

At true closed extension points, every omission must be handled or deliberately classified. Generic downstream analyses should not grow AST classifications at all; they consume canonical CFG/type/place/effect evidence instead.

14. Adding a language feature

For a new syntax construct, in order:

token — a keyword or token kind, if the syntax needs one.
AST node — the struct, its family marker, and forEachChild.
parser — build the node, with recovery.
resolver — scopes and bindings.
typechecker — the type rule, and publish whatever later phases will need.
CFG — only if the control-flow shape is genuinely new.
analysis — implement required AnalyzeFlow and BuildEffects methods when the construct changes flow or storage behavior.
THIR/MIR — only if no existing lowering shape can represent it.

Steps 1–6 are unavoidable: where a name lives and what types are legal is the feature. Step 7 is what buys you definite initialization, ownership, liveness, drops and usage for free — they consume operations and never learn your construct exists.

docs/compiler-architecture.md defines why these boundaries exist; docs/compiler-framework/change-paths.md gives the concrete edit path and the guard at each true extension point.

15. Glossary

Term	Meaning
NodeID	Canonical source identity; function syntax is owned by `FunctionID` plus local preorder
SymbolID	Stable identity of one declaration
SiteID	`{Block, Index}` — one ordered program point in a CFG
Place	Storage: a root binding (or a temporary) plus projections
Artifact	A phase's published output, owned by exactly one phase
Prelude	Implicitly imported module every other module depends on
Contract	A test that forces an explicit decision for every node kind
Validator	A boundary check on an artifact's shape, reported as a compiler bug

16. Where to look when

Question	Start here
Why is my program rejected?	`internal/diagnostics/codes.go`, then the phase that owns the code
How does phase ordering work?	`internal/pipeline/pipeline.go`, `advanceModulePhase`
What does the typechecker publish?	`internal/semantics/typechecker/evidence.go`, then `internal/ir/thir`
Why is a value moved/dropped here?	`internal/semantics/analysis`, then `internal/ir/mir`
What does the backend emit for X?	`internal/backend/llvm/emitter.go`
How do I add a node kind safely?	`docs/compiler-architecture.md`, then the owning syntax boundary

Navigation Menu

Repository navigation

Code tour: from `.peep` to an executable

1. The thirty-second version

2. Package map

3. Entry: command → driver → pipeline

4. Module loading

5. The phase ladder

What each phase produces

6. Front end: text → AST

7. Semantic analysis

8. Control-flow graph

9. Effects: the semantic operation stream

10. The analyses that consume it

11. Lowering: THIR + CFG → MIR

12. Backend and linking

13. Guardrails

14. Adding a language feature

15. Glossary

16. Where to look when

FilesExpand file tree

Code-tour.md

Latest commit

History

Code-tour.md

File metadata and controls

Code tour: from .peep to an executable

1. The thirty-second version

2. Package map

3. Entry: command → driver → pipeline

4. Module loading

5. The phase ladder

What each phase produces

6. Front end: text → AST

7. Semantic analysis

8. Control-flow graph

9. Effects: the semantic operation stream

10. The analyses that consume it

11. Lowering: THIR + CFG → MIR

12. Backend and linking

13. Guardrails

14. Adding a language feature

15. Glossary

16. Where to look when

Code tour: from `.peep` to an executable