Fast, lazy font opening across source formats

[kostyafarber]

Goal

Open production font sources quickly enough that Shift feels immediate, while preserving complete authoring semantics. The target is under 300 ms from the file chooser returning to the first useful glyph grid content on the representative dogfood fonts.

The whole font does not need to be materialized before the document appears. After the directory and first visible glyph projections are ready, complete authored glyphs can continue loading into the workspace store in the background.

Representative repro:

/Users/kostyafarber/repos/shift-dogfood/sources/fraunces/Fraunces.designspace

This applies across Designspace, UFO, Glyphs/Glyphspackage, binary fonts, and .shift packages. It is not a Designspace-only optimization.

Agreed API boundary

The native caller remains:

FontWorkspace::open(source_path, store_path)

FontLoader::read_font(path) -> Font remains the eager, full-fidelity reader and correctness backstop. It does not have to be the fast document-open path.

Do not add public OpenedFont, Definition, DocumentIndex, openIndex, hydration-session, readiness, request-generation, or location-keyed preview APIs. Do not make shift_font::Font an ambiguously partly loaded public value. Staged native state belongs inside FontWorkspace and its internal source reader.

The renderer continues to use its existing Font model and demand APIs:

• requestGlyphViews(glyphIds) for compact glyph projections;
• loadGlyph(glyphId) and loadGlyphs(glyphIds) for complete authored glyphs;
• the grid normally consumes projections through useGlyphViews;
• FontStore owns resident glyph records, projections, authored layers, variation models, and their signals;
• WorkspaceEditCoordinator orders batched reads and edits.

The implementation behind these operations may become lazy. Callers should not manage hydration sessions or global readiness flags.

Opening sequence

1. Capture workspace-owned immutable backing for the complete imported source closure.
2. Read and persist the font directory:
   • metadata and metrics;
   • axes, mappings, and labels;
   • sources, source metrics, and named instances;
   • glyph order, IDs, names, unicodes, and source/layer addressing metadata.
3. Load one batched set of compact projections for the visible and overscanned grid glyphs, including required component dependencies.
4. Reveal a useful document and glyph grid.
5. Continue materializing in the background, off the renderer and Electron main threads:
   • finish globally useful lightweight structure such as the component-reference graph;
   • load complete authored glyph and layer bodies into SQLite in bounded batches;
   • reach the same complete workspace state as the eager reader.

Axis scrubbing evaluates resident compact projection models synchronously in TypeScript. It must not issue location-keyed native requests.

Background loading and foreground commands

Background loading is opportunistic, not a separate competing ownership model.

• Background work and user commands share one coordinator and single-flight work per glyph/resource.
• Visible projections and foreground commands have priority over speculative background batches.
• If a command needs unloaded information, it promotes and awaits only its prerequisites before mutating. It must never guess from partial data.
• For example, deleting a selection may require the complete component-reference index, but should not require every unrelated outline to be decoded first.
• Materialization batches and edit transactions are ordered so a late import result cannot overwrite an edit or resurrect a deleted glyph.
• Pending work for deleted or replaced entities is cancelled or rejected before commit.

These are internal residency semantics. The normal renderer API remains Font, projections, authored glyph loading, and signals.

Workspace-owned lazy backing

Once open succeeds, later lazy reads must not depend on the user's original source still existing or remaining unchanged. The original path is provenance and an external-change reference only.

The workspace must own an immutable raw-byte snapshot of the complete source closure:

• the single input file for TTF/OTF and monolithic Glyphs;
• the whole UFO or Glyphspackage directory;
• the Designspace file and every referenced UFO directory;
• preservation material such as layers, features, kerning, libs, data, images, and unknown files.

Capturing this snapshot must not require parsing or normalizing every glyph. Prefer copy-on-write clones or reflinks where available, with a correctness-preserving fallback. Do not use hardlinks: an in-place change to the original would also change the workspace backing. The exact cross-platform capture mechanism remains an implementation decision for this ticket.

SQLite is the canonical mutable workspace. The source snapshot is immutable backing for records that have not yet been materialized.

Reader-level work

The lazy architecture does not replace reader optimization. Each source format should avoid work not needed for the current phase.

• Glyphspackage: read top-level metadata and ordering first; decode glyph files on demand or in background batches.
• UFO: read root metadata and contents.plist first; decode .glif bodies on demand. Reuse each unique UFO parse/index.
• Designspace: read axes and source descriptors first; index referenced UFOs without eagerly decoding all geometry.
• Binary fonts: retain table-backed/random-access data where possible and decode glyph bodies on demand.
• .shift: open its directory/store representation without rebuilding an intermediate complete font unnecessarily.

The previous whole-FontIndex rebuild per inserted layer has already been removed and is not the remaining architectural direction. Continue profiling parser conversion, component resolution, snapshot capture, initial SQLite writes, and background materialization.

Correctness and full fidelity

• Lazy open and eager FontLoader::read_font must produce equivalent authored font semantics after full materialization.
• Save and export must include unloaded glyphs and preservation data. They may force completion initially, but may never silently skip data.
• Global operations may await a larger prerequisite barrier when correctness requires it.
• Glyph/source/layer order and identities, default source, axes and mappings, components, metrics, interpolation, features, kerning, libs, and binaries must remain intact.
• Imported source bytes must remain available to the workspace after the original is moved, edited, or deleted.

Measurement

Measure these milestones separately from the native file dialog:

1. immutable source backing captured;
2. font directory ready;
3. first visible compact projection batch ready;
4. first useful grid painted;
5. global structural index complete;
6. full authored workspace materialized in SQLite.

Preliminary eager-path profiling found roughly 1.0–1.2 s for Fraunces after bulk-construction improvements, with smaller representative Glyphs, TTF, and UFO inputs around 126 ms, 149 ms, and 307 ms respectively. These are investigation numbers, not stable benchmarks or CI thresholds.

Acceptance criteria

• The representative large fonts target under 300 ms from chooser return to useful grid content, with before/after phase measurements recorded in the PR.
• Initial open does not decode or insert every full glyph body before revealing the document.
• Visible and overscanned glyph projections load in batches with component dependencies.
• Background materialization proceeds in bounded work without making scrolling, scrubbing, or editing choppy.
• A foreground edit issued before background completion promotes its prerequisites and commits correctly; a late background batch cannot overwrite it.
• Full background materialization produces the same semantic font as the eager reader.
• Save/export while materialization is incomplete produces a complete result.
• Lazy reads continue to work after the original source is changed, moved, or deleted.
• Performance coverage uses profiling fixtures or benchmarks rather than flaky wall-clock assertions in normal CI.

[kostyafarber]

Scope clarification from dogfooding: this issue is about cold first load, measured from the file chooser returning a path until the renderer receives a usable workspace/document snapshot. It should include backend parsing, IR construction, component resolution, and initial store materialization.

Do not include completion of glyph-grid preview hydration in that measurement. Preview loading/scrolling has its own pipeline and can continue after the document is usable.

[kostyafarber]

2026-07-18 stress profile: 1.2 GB UFO

Profiled TT2020StyleF-Regular.ufo (1.2 GB, 7,320 GLIFs, approximately 30.9 million points and 3.2 million contours) in a release build.

Eager reader findings

Initial baseline:

• NORAD parse: 6.86 s
• UFO → Shift conversion: 35.0 s
• backend total: 43.5 s
• normalized SQLite materialization was still unfinished after 5 minutes

After enabling NORAD's optional Rayon parser, parallelizing ordered UFO conversion, batching entropy reads for generated IDs, and storing short IDs inline:

• NORAD parse: approximately 1.2–1.5 s
• UFO → Shift conversion: approximately 6.7–8.6 s
• complete in-memory Font: approximately 9.5–13.0 s depending on run

This confirms NORAD is not a fixed 6.86-second floor. Reader-level optimization is worthwhile and should remain part of this ticket, but an eager full Font cannot meet the 300 ms first-useful-grid target for a 1.2 GB source.

Working-store finding

The current initial population expands every point into one SQLite row with a TEXT primary key plus order indexes. On the full fixture, the run was stopped after nearly 12 minutes:

• store population was still running
• WAL had reached 2.4 GB
• process RSS was still around 3.4 GB
• disabling WAL durability experimentally did not materially improve row construction; B-tree maintenance dominates

Removing redundant indexes that are already covered by the leftmost prefix of unique order indexes improved a 33 MB proxy store from about 2.09 s to 1.71 s, but this does not change the scaling shape.

A measurement-only packed-layer prototype stored one complete binary payload per glyph layer:

• 33 MB proxy: 159 ms, 31.9 MB database (versus 1.71 s normalized)
• 1.2 GB UFO: 5.57 s, 1.157 GB database
• full read + packed persistence: 19.7 s

The prototype was removed after measurement; it is not a proposed codec commitment.

Memory finding

Immediately after constructing the full in-memory font, RSS was approximately 7.87 GB; after transient allocator pages were reclaimed it remained around 5.73 GB.

A measurement-only run without the duplicate global point/contour identity sets reduced font construction from 12.55 s to 8.68 s, but steady RSS only fell to about 5.46 GB. The uniqueness invariant should be retained and its index compacted; the dominant live memory is the authored 31 million-point body itself. A genuinely small open footprint therefore requires lazy glyph/layer materialization.

Consequences for this ticket

1. Keep FontLoader::read_font as the eager correctness backstop and land the safe reader improvements independently.
2. The under-300 ms product path must stop after directory metadata and the visible/overscanned glyph batch; it cannot await full source decoding.
3. Decouple semantic edit granularity from working-store representation. FontChangeSet may remain granular while persistence snapshots complete touched layer aggregates.
4. Investigate packed per-layer authored payloads for recovery, while retaining separately queryable structural indexes such as component references. Do not create 31 million indexed point rows during initial open.
5. Save/export/global operations must still force or coordinate complete materialization where required.
6. Add this 1.2 GB UFO as a manual stress fixture and keep ordinary CI performance coverage fixture-sized and non-wall-clock-flaky.

[kostyafarber]

Agreed working-store and Cmd+S model

The working store should not normalize authored outlines into one SQL row per point. Keep SQLite as the canonical mutable workspace, but use two complementary representations.

Packed authored layer payloads

Persist one versioned binary payload per complete authored glyph layer. The payload owns the layer's:

• advance and optional height;
• contours, points, point metadata, and stable identities;
• components and transforms;
• anchors and guidelines;
• layer-owned preservation/lib data.

Semantic edit granularity and persistence granularity are intentionally different. FontChangeSet continues to describe the precise authored operation, while the store writes the resulting post-edit snapshot of each touched layer. A point drag therefore replaces one layer payload, not thousands of SQL point rows and not the whole font.

The binary codec is an internal working-store detail, versioned with the store schema. It is not the .shift authoring schema and must not leak into bridge or renderer types.

Separately queryable structural tables

Retain ordinary indexed SQL tables for information needed without decoding every glyph body:

• glyph identity, name, Unicode values, and order;
• layer identity, glyph ownership, source ownership, and order;
• axes, mappings, sources, source locations, metrics, and named instances;
• component references from owning glyph/layer to base glyph;
• other proven font-wide reference indexes needed by global commands.

These indexes are derived and updated in the same transaction as the owning packed layer snapshot. They let Shift answer questions such as “which glyphs reference this base glyph?”, “which layers belong to this source?”, and “what is visible in the glyph directory?” without loading every outline.

Do not create redundant point/contour SQL records merely to make the payload queryable. Add a structural index only when a real global operation requires it.

Edit persistence versus Cmd+S

A successful editor command persists before it is considered committed:

1. apply the intent to the next in-memory Font;
2. write the touched layer payloads and derived structural-index changes in one SQLite transaction;
3. mark the workspace revision dirty;
4. publish the new live font.

Thus editing one glyph immediately writes that glyph's touched layer payload to the recovery store. It does not wait for Cmd+S. A crash before Cmd+S can still resume the dirty workspace from SQLite.

Cmd+S has a different responsibility: create a complete, self-consistent user-authored .shift package and mark the current workspace revision saved. .shift is currently one atomically replaced ZIP, so Cmd+S must not patch only one glyph entry in place.

Initially, Cmd+S may force any still-unmaterialized data to complete, then stream a complete deterministic package into a temporary ZIP and atomically replace the destination. A later optimized writer may copy unchanged entries and regenerate only dirty glyph documents while producing the new archive, but the observable result remains a complete atomic package.

For an imported UFO/Designspace/Glyphs source, the first Save As must likewise produce the complete .shift; it may not omit unloaded glyphs. For a future loose-directory .shift container, narrow per-glyph file writes could be considered separately, but that is not the current ZIP contract.

Acceptance additions

• Opening and ordinary editing do not require point-per-row SQLite materialization.
• A one-layer edit rewrites only that layer payload and its affected structural-index rows in the working store.
• Component-dependant and glyph-directory queries work without decoding every layer payload.
• Resume reconstructs the exact live font from packed payloads plus relational metadata/indexes.
• Cmd+S during or after lazy loading always creates a complete atomic .shift package; it either coordinates/streams unloaded content correctly or explicitly waits for completion.
• Store codec compatibility and corruption errors are typed and versioned independently of the .shift schema.

[kostyafarber]

The renderer API, first-complete-grid-frame, and scrolling direction described here have been refined in #159. In particular, #159 replaces the requestGlyphViews / loadGlyph caller model with intent-specific loadGlyphView and loadEditableGlyph acquisition, retained synchronous objects, and active/pending pinned Grid frames. This issue remains authoritative for native lazy source opening, immutable source backing, background materialization, and the under-300 ms first-useful-content goal.