Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0ec2fee
docs: start milestone v1.23 io boundary
gabewillen May 4, 2026
f0013c3
feat: add io loading boundary
gabewillen May 4, 2026
e56e370
fix: close io boundary review gaps
gabewillen May 4, 2026
e9739e4
fix: reset tensor loader after io failure
gabewillen May 4, 2026
1e747c2
docs(roadmap): add gap closure phase 211
gabewillen May 4, 2026
a44499f
docs(211): execute phase 211 verification artifact backfill
gabewillen May 4, 2026
d1fd1a2
docs(audit): re-audit v1.24 after phase 211 backfill (passed)
gabewillen May 4, 2026
217944b
docs(211): add wave field to phase plan frontmatter (linter)
gabewillen May 4, 2026
6358b6a
chore: archive v1.24 milestone files
gabewillen May 4, 2026
438d200
chore: remove REQUIREMENTS.md for v1.24 milestone
gabewillen May 4, 2026
5e64d62
feat: add mmap loading strategy
gabewillen May 4, 2026
c0774df
fix: address io mmap PR review comments
gabewillen May 4, 2026
92712ce
merge origin/main into issue 60 branch
gabewillen May 5, 2026
12e8f1f
fix: reject mmap spans beyond eof
gabewillen May 5, 2026
8aad365
fix: close mmap review follow-ups
gabewillen May 5, 2026
70c622b
fix: simplify mapped load done transition
gabewillen May 5, 2026
278ccf5
fix: require mmap map completion callback
gabewillen May 5, 2026
7ba0155
fix(io/mmap,model/tensor): retain slot on unmap failure; reject legac…
gabewillen May 5, 2026
22a240b
fix: preserve mmap metadata and partial cleanup state
gabewillen May 5, 2026
7de907a
fix: preserve mmap bindings on storage rebind errors
gabewillen May 5, 2026
172df14
fix: reject bind tensor on mmap resident tensors
gabewillen May 5, 2026
111cd6c
fix: align v1.24 planning state counters
gabewillen May 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .planning/MILESTONES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,50 @@
# Project Milestones: EMEL

## v1.24 I/O Mmap Loading Strategy (Shipped: 2026-05-04)

**Phases completed:** 8 phases, 8 plans, 0 tasks

Comment thread
gabewillen marked this conversation as resolved.
**Key accomplishments:**

- Established the canonical `src/emel/io/mmap` Stateforward.SML actor with component-local
context, events, guards, actions, errors, and `emel::io::mmap::sm` ownership.

- Modeled mmap request/platform/file/offset/length/layout validation and unsupported-platform
rejection through explicit guards and transitions before any mapping attempt.

- Added real `open`+`mmap`+`munmap` paths under `#if defined(_WIN32)` selection, a
fixed-capacity slot pool (`EMEL_IO_MMAP_MAX_MAPPINGS = 256`), `event::release_mapping`
as the actor-owned unmap surface, and a deterministic mmap error taxonomy.

- Added `event::request_mapped_load` / `event::release_mapped_load`,
`lifecycle::mmap_resident`, and `sm(emel::io::mmap::sm*)` injection on `model/tensor`,
preserving tensor-owned load/bind/evict/residency orchestration with zero handle state in
tensor.

- Kept `model/loader`, maintained benchmark, paritychecker, and embedded-probe lanes off
actor internals; mmap reporting flows through public tensor surfaces only.

- Added doctest proof of supported and rejection mmap behavior through `process_event(...)`
and three new domain-boundary script rules guarding scope and ownership.

- Aligned README, README template, parity roadmap, and architecture docs with the
implemented mmap path; refreshed `snapshots/bench/benchmarks.txt` for `encoder_spm`
and `encoder_wpm` via maintained scoped `scripts/bench.sh --snapshot --compare --update`;
removed the Phase 204 transitional bench-regression override.

- Backfilled missing per-phase VERIFICATION.md artifacts for Phases 208, 209, and 210;
closed the v1.24 audit's 3-source cross-reference gap.

**Audit:** Final source-backed audit passed with 13/13 active requirements satisfied
(MMAP-01..03, TIO-01..03, PLAT-01, LIFE-01, ERR-01, VAL-01..04). Closing
`EMEL_QUALITY_GATES_SCOPE=full scripts/quality_gates.sh` exit 0 (no override; total 432s).

**Known deferred items at close:** Concrete read/copy/async/device strategies remain
follow-on work below the `emel/io` boundary. The previously deferred non-v1.23 quick
task and four optimization todos remain tracked in `.planning/STATE.md`.

---

## v1.23 I/O Loading Strategy Boundary (Shipped: 2026-05-04)

**Phases completed:** 7 phases, 7 plans, 0 tasks
Expand All @@ -8,12 +53,16 @@

- Established `src/emel/io` as a Stateforward.SML loading-boundary module with fail-closed strategy
scaffolding and public aliases.

- Added explicit IO request/result/error events and tensor IO load effects without moving tensor
residency ownership.

- Modeled IO strategy policy and rejection through explicit guards and transitions, with no hidden
action/detail routing.

- Integrated model-loader orchestration with the public IO actor boundary while keeping maintained
tool lanes off actor internals.

- Repaired v1.23 closeout proof with public test surfaces, stronger guardrails, generated docs truth, and passing scoped gates.
- Closed v1.23 closeout tech debt with planning-state truth, tensor context cleanup, IO benchmark-state markers, and passing gates.

Expand Down
55 changes: 40 additions & 15 deletions .planning/PROJECT.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ before widening API surface or model scope.

## Current State

Current milestone: none open
Current milestone: `v1.24 I/O Mmap Loading Strategy`

Latest shipped milestone: `v1.23 I/O Loading Strategy Boundary`

Expand All @@ -25,7 +25,27 @@ Status: `v1.23` shipped on 2026-05-04 after final source-backed audit passed. Th
owner of tensor load, bind, evict, and residency semantics, and `model/loader` as the orchestrator
across those public actor surfaces.

Current planning focus: start the next milestone when ready.
Current planning focus: implement the issue #61 mmap strategy path beneath the v1.23 I/O
boundary.

## Current Milestone: v1.24 I/O Mmap Loading Strategy

**Goal:** Add a dedicated `io/mmap` Stateforward.SML strategy actor under `src/emel/io` so
tensor-owned model loading can request memory-mapped residency through the existing I/O boundary
without moving tensor lifecycle ownership out of `model/tensor` or adding read/copy/async strategy
behavior.

**Source:** GitHub issue #61, "Add io/mmap state machine for tensor-backed model loading"

**Target features:**
- Dedicated `src/emel/io/mmap` machine, events, guards, actions, context, errors, and public
aliases for mmap-backed tensor loading.
- Tensor-to-I/O integration that lets `model/tensor` request mmap-backed residency while retaining
tensor-owned load, bind, evict, and residency semantics.
- Explicit mmap success, unsupported, validation, and platform/resource failure outcomes surfaced
deterministically through events and states.
- Maintained tests, docs, lint snapshots, benchmark snapshots, benchmark outputs, and model
artifacts updated from maintained commands when required.

## Previous Shipped Milestone: v1.23 I/O Loading Strategy Boundary

Expand Down Expand Up @@ -347,14 +367,14 @@ truth anchor and without broadening into generic Liquid-family support.

### Active

- v1.22 cuts weight-loading ownership from `src/emel/model/weight_loader` into
`src/emel/model/tensor` while preserving existing model-loading behavior.
- v1.22 updates `src/emel/model/loader` so bulk model loading orchestrates tensor-owned behavior
instead of treating `load_weights_fn` as the long-term architecture seam.
- v1.22 retires or explicitly bounds the old `model/weight_loader` path so the codebase does not
retain a second tensor-residency owner under a new name.
- v1.22 prepares, but does not implement, the future `emel/io` strategy layer below tensor
ownership.
- v1.24 adds a dedicated `src/emel/io/mmap` Stateforward.SML strategy actor for mmap-backed
tensor loading.
- v1.24 integrates mmap-backed residency requests through the existing tensor-to-I/O boundary while
`model/tensor` remains the tensor lifecycle and residency owner.
- v1.24 models mmap support, validation, success, and failure as explicit guard/state/event
behavior without hiding runtime strategy choice in actions or detail helpers.
- v1.24 keeps staged read/copy, device-specific loading, cooperative async loading, new model
families, and broad public API expansion out of scope.

### Validated

Expand Down Expand Up @@ -521,10 +541,11 @@ tooling that publishes through canonical compare/benchmark contracts without sha
`v1.18` and `v1.19` provide the parity and benchmark dependency manifests that v1.21 now consumes
from the top-level quality-gate orchestration. `v1.21` shipped from issue #58 and did not weaken
mandatory validation or change benchmark/parity semantics. `v1.22` shipped from issue #59 and made
`model/tensor` the canonical owner of tensor load, bind, evict, and residency behavior while
keeping concrete I/O strategy work deferred. `v1.23` is open from issue #60 and adds the missing
`emel/io` orchestration boundary under tensor-owned residency without implementing concrete mmap,
read/copy, staged, chunked, or asynchronous strategy machines.
`model/tensor` the canonical owner of tensor load, bind, evict, and residency behavior. `v1.23`
shipped from issue #60 and added the missing `emel/io` orchestration boundary under tensor-owned
residency while explicitly deferring concrete mmap, read/copy, staged, chunked, and asynchronous
strategy machines. `v1.24` starts issue #61 and is the first concrete strategy milestone: mmap
only, under `src/emel/io`, with tensor residency still owned by `model/tensor`.

## Constraints

Expand Down Expand Up @@ -558,11 +579,15 @@ read/copy, staged, chunked, or asynchronous strategy machines.
- **I/O boundary scope**: `v1.23` creates the `emel/io` module and tensor-to-I/O contract only. It
must not implement concrete mmap, read/copy, staged/chunked, device-specific, or cooperative async
loading strategies; those belong in follow-on milestones such as issue #61.
- **Mmap strategy scope**: `v1.24` implements only the mmap strategy behind `emel/io`. It must not
add staged read/copy, chunked, device-specific, or cooperative async loading behavior, and must
not move tensor residency lifecycle ownership out of `model/tensor`.

## Key Decisions

| Decision | Rationale | Outcome |
|----------|-----------|---------|
| Start v1.24 from GitHub issue #61 as the `io/mmap` loading strategy milestone | v1.23 established the `emel/io` strategy boundary and explicitly deferred concrete mmap behavior; issue #61 is the next narrow strategy path to land beneath tensor-owned residency | - Pending |
| Start v1.23 from GitHub issue #60 as the `emel/io` boundary milestone | v1.22 moved tensor residency ownership into `model/tensor`; the next architecture step is the explicit I/O strategy seam beneath tensor-owned residency before concrete mmap or staged strategy work lands | Phase 203 closeout cleanup |
| Start v1.22 from GitHub issue #59 as the weight-loading ownership cutover | `model/tensor` owns individual tensor lifecycle state while `model/weight_loader` still owns bulk residency transition planning; the next runtime architecture milestone should remove that split before adding future I/O strategy work | ✓ Shipped |
| Start v1.21 from GitHub issue #58 as quality-gate selective runner optimization | v1.18 and v1.19 added parity and benchmark dependency manifests; the next milestone should cash in that structure at the mandatory gate-orchestration level without weakening conservative fallback behavior | ✓ Shipped |
Expand Down Expand Up @@ -619,4 +644,4 @@ This document evolves at phase transitions and milestone boundaries.
4. Update Context with current state

---
*Last updated: 2026-05-04 during Phase 203 closeout cleanup for v1.23*
*Last updated: 2026-05-04 after starting v1.24 I/O mmap loading strategy milestone*
82 changes: 72 additions & 10 deletions .planning/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,79 @@

## Milestones

- [x] **v1.23 I/O Loading Strategy Boundary** — shipped 2026-05-04; archived in
`.planning/milestones/v1.23-ROADMAP.md`, requirements in
`.planning/milestones/v1.23-REQUIREMENTS.md`, audit in
`.planning/milestones/v1.23-MILESTONE-AUDIT.md`, and phase artifacts in
`.planning/milestones/v1.23-phases/`.
- ✅ **v1.0 EMEL Llama-68M Generation Slice** — shipped 2026-03-08
- ✅ **v1.1 EMEL Llama-68M Generation Benchmark** — shipped 2026-03-11
- ✅ **v1.2 Flash Attention** — shipped 2026-03-22
- ✅ **v1.3 ARM Flash Optimizations** — shipped 2026-03-22
- ✅ **v1.4 Full Vectorized Quantized Kernels** — shipped 2026-03-25
- ✅ **v1.5 Full ARM Quantized Path** — shipped 2026-03-27
- ✅ **v1.6 Qwen3-0.6B Parity And Benchmark** — shipped 2026-03-30
- ✅ **v1.7 Generator Prefill Submachine Decomposition** — shipped 2026-03-30
- ✅ **v1.8 Truthful Qwen3 E2E Embedded Size** — shipped 2026-04-02
- ✅ **v1.9 Liquid LFM2.5-1.2B Thinking ARM Slice** — shipped 2026-04-02
- ✅ **v1.11 TE-75M GGUF Trimodal Embedding Runtime** — shipped 2026-04-15
- ✅ **v1.12 Pluggable Reference Parity Bench Architecture** — shipped 2026-04-18
- ✅ **v1.13 Pluggable Generative Parity Bench** — shipped 2026-04-21
- ✅ **v1.14 Benchmark Variant Organization** — shipped 2026-04-21
- ✅ **v1.15 ARM Sortformer Diarization GGUF Slice** — shipped 2026-04-25
- ✅ **v1.16 ARM Whisper GGUF Parity And Performance** — shipped 2026-04-28
- ✅ **v1.17 Text Generator Domain Alignment** — shipped 2026-04-30
- ✅ **v1.18 Parity Tool Boundary Refactor** — shipped 2026-05-01
- ✅ **v1.19 Benchmark Tool Pluggable Runner Refactor** — shipped 2026-05-01
- ✅ **v1.20 SML Dependency And Namespace Migration** — shipped 2026-05-02
- ✅ **v1.21 Quality Gate Selective Runner Optimization** — shipped 2026-05-02
- ✅ **v1.22 Weight Loading Ownership Cutover** — shipped 2026-05-03
- ✅ **v1.23 I/O Loading Strategy Boundary** — shipped 2026-05-04
- ✅ **v1.24 I/O Mmap Loading Strategy** — shipped 2026-05-04 (Phases 204-211)

## Current Work
## Phases

No active milestone is open. Start the next milestone with `$gsd-new-milestone`.
<details>
<summary>✅ v1.24 I/O Mmap Loading Strategy (Phases 204-211) — SHIPPED 2026-05-04</summary>

## Deferred Items
- [x] Phase 204: Mmap Strategy Component Boundary (1/1 plans) — completed 2026-05-04
- [x] Phase 205: Mmap Validation and Platform Gating (1/1 plans) — completed 2026-05-04
- [x] Phase 206: Mapped Descriptor, Errors, and Lifetime (1/1 plans) — completed 2026-05-04
- [x] Phase 207: Tensor-Owned Mmap Integration (1/1 plans) — completed 2026-05-04
- [x] Phase 208: Public Runtime and Evidence Surfaces (1/1 plans) — completed 2026-05-04
- [x] Phase 209: Behavior Tests and Scope Guardrails (1/1 plans) — completed 2026-05-04
- [x] Phase 210: Publication and Maintained Artifact Updates (1/1 plans) — completed 2026-05-04
- [x] Phase 211: Phase Verification Artifact Backfill (1/1 plans) — completed 2026-05-04 (gap closure)

Previously acknowledged non-v1.23 quick task and optimization todos remain tracked in
`.planning/STATE.md`.
Archive:
- `.planning/milestones/v1.24-ROADMAP.md`
- `.planning/milestones/v1.24-REQUIREMENTS.md`
- `.planning/milestones/v1.24-MILESTONE-AUDIT.md`
- `.planning/milestones/v1.24-phases/{204..210}-*` (Phase 211 backfill artifacts live alongside their parent phase dirs)

</details>

<details>
<summary>✅ v1.23 I/O Loading Strategy Boundary (Phases 197-203) — SHIPPED 2026-05-04</summary>

Archive:
- `.planning/milestones/v1.23-ROADMAP.md`
- `.planning/milestones/v1.23-REQUIREMENTS.md`
- `.planning/milestones/v1.23-MILESTONE-AUDIT.md`
- `.planning/milestones/v1.23-phases/`

</details>

### 📋 Next Milestone

Define the next milestone (v1.25 or higher) via `$gsd-new-milestone`. Concrete read/copy,
async, and device loading strategies remain deferred follow-on work below the
`emel/io` boundary.

## Progress

| Phase | Milestone | Plans Complete | Status | Completed |
|-------|-----------|----------------|--------|-----------|
| 204. Mmap Strategy Component Boundary | v1.24 | 1/1 | Complete | 2026-05-04 |
| 205. Mmap Validation and Platform Gating | v1.24 | 1/1 | Complete | 2026-05-04 |
| 206. Mapped Descriptor, Errors, and Lifetime | v1.24 | 1/1 | Complete | 2026-05-04 |
| 207. Tensor-Owned Mmap Integration | v1.24 | 1/1 | Complete | 2026-05-04 |
| 208. Public Runtime and Evidence Surfaces | v1.24 | 1/1 | Complete | 2026-05-04 |
| 209. Behavior Tests and Scope Guardrails | v1.24 | 1/1 | Complete | 2026-05-04 |
| 210. Publication and Maintained Artifact Updates | v1.24 | 1/1 | Complete | 2026-05-04 |
| 211. Phase Verification Artifact Backfill | v1.24 | 1/1 | Complete | 2026-05-04 |
Loading