Store glyph layers lazily and stream foreign imports

[kostyafarber]

Summary

• define canonical shift.glyph-layer.v1 framing in Rust and TypeScript with strict validation, shared goldens, property coverage, and benchmarks
• store each authored layer as one independently addressable SQLite BLOB with exact stored/decoded bounds, decoded BLAKE3, and independent none | zstd.v1 compression
• resume workspaces directory-first with explicit bounded acquisition, atomic batch hydration, and safe eviction
• stream TTF/OTF, UFO, and Designspace imports through bounded glyph/layer batches with overlapped parsing, parallel packing/hash/compression, and one transactional SQLite writer
• enumerate every compiled-font maxp glyph ID, preserve source TrueType quadratics, and derive stable positional contour/point IDs for equivalent compiled imports
• validate complete layer replacement batches before mutating uniquely owned fonts in place, while preserving COW snapshots and atomic failure behavior
• add disposable import pragmas, transactionally deferred secondary indexes, durable finalization, rollback-on-drop, and sibling staged-store installation that preserves any previous destination

Storage contract

shift.glyph-layer.v1 remains the immutable canonical inner payload. SQLite stores one complete independent layer as either raw canonical bytes or one dictionary-free Zstandard frame. Zstd level 1 is the current write policy; writers retain it only when smaller. Every row records the inner format, compression kind, exact stored and decoded lengths, and a 32-byte BLAKE3 of the canonical bytes.

Reads bound stored and decoded sizes before allocation, reject truncated/trailing frames, require exact decoded length and hash, then run strict canonical decoding and cross-check directory/component facts. Mixed compressed/uncompressed rows and transitions are supported without touching unrelated layers.

Final architecture review

A second full origin/main...HEAD review confirmed the crate boundaries and storage-validation ordering, then drove a focused cleanup:

• preserve each non-default Designspace master's standard metrics by translating from that UFO's own metric identities
• make ShiftStore::load_glyph_layers the single count/decoded-byte planner used by acquisition and full materialization
• stage foreign imports through explicit Staging -> Durable -> Published transitions before atomic installation and parent-directory sync
• replace five identity-set out-parameters with ReplacementIdentities, retain previous layers through Arc, and move validated sets into the final index
• share Designspace source registration and convert GLIF geometry directly without a throwaway Glyph or deep layer clone
• remove denormalized glyph_layers.name, join canonical glyph names, and preserve authored glyph order on rename
• preserve relative UFO non-default layer order when public.default was not listed first
• tighten codec string estimates, hostile allocation bounds, large data checks, diagnostics, and deterministic malformed-input coverage

The remaining review nits—renaming the established public outline CodecError, private TS Header naming, and tiny binary-helper deduplication—were deliberately left alone because their churn exceeds their correctness or boundary value.

Scale evidence

Release profiles on a 12-thread Ryzen 5 5500U with eight Rayon workers. These are observational runs rather than CI timing gates.

Source Han Sans VF

• 65,535 / 65,535 glyph IDs and layers imported; 44,853 Unicode rows
• directory discovery/publication: about 83 ms
• overlapped parser/packer/compressor/SQLite pipeline: about 1.77 s
• complete durable import and directory materialization: about 2.46 s
• 379,332,924 canonical bytes -> about 109.3 MB stored payload
• complete SQLite document: 175,726,592 bytes (167.6 MiB), down from 449,122,304 bytes uncompressed
• peak RSS: about 253 MiB
• native directory-only resume: about 0.41 s with zero loaded payloads
• one-layer acquisition: 0.24 ms total, including a 0.14 ms bounded read/decompress/hash/decode
• one-layer eviction: 0.02 ms

Before in-place validated hydration, the same acquisition/eviction path took about 128 ms / 53.8 ms because it copied and synchronously dropped the complete 65,535-glyph font state.

Roboto Flex Designspace

• 13 axes, 85 sources, 948 glyphs, 80,580 authored layers, and 83,215 component rows
• complete directory discovery: about 153–164 ms
• pipeline: about 1.63–1.97 s; complete import about 2.26–2.62 s depending source filesystem cache
• 64,343,071 canonical bytes -> about 36.38 MB stored payload
• complete SQLite document: about 75.5 MB (72.0 MiB)
• native resume: about 0.20 s with zero loaded payloads
• one glyph across 85 masters: 2.50 ms to acquire and 0.66 ms to evict, down from 85.3 ms / 29.3 ms

Direct Roboto Flex UFO

• 948 glyphs and 953 foreground/background layers
• pipeline: about 20 ms; complete import about 65 ms
• 739,096 canonical bytes -> about 395 KB stored payload
• complete SQLite document: about 1.42 MB
• native resume: about 7.9 ms
• two-layer acquisition: about 0.69 ms

Verification

• cargo test --workspace --exclude shift-bridge
• cargo clippy --workspace --exclude shift-bridge --all-targets -- -D warnings
• cargo clippy -p shift-bridge --all-targets -- -D warnings
• cargo fmt --all -- --check
• glyph-codec TypeScript: 34 Vitest tests and isolated tsc --noEmit
• repository pnpm format:check and pnpm lint:check
• SQLite integrity and stored/decoded/hash checks on Source Han, Roboto Flex Designspace, and direct UFO databases
• python3 scripts/context-drift-check.py (still reports the existing unrelated app-documentation errors)

Commit hooks passed all applicable Rust checks. The unavailable worktree-local JS hooks were run manually against the main checkout's pinned tools without creating node_modules symlinks.

Follow-ups

• publish foreign-source directory/preview readiness independently of background canonical conversion
• add app-facing progress, cancellation, visible-glyph priority, and staged validation beyond the now-atomic workspace installation
• replace native-open empty GlyphLayer placeholders with a compact complete directory after the resident Slug grid work settles
• preserve complete compiled-font variation sources rather than only default-location geometry
• rerun the externally configured TT2020 corpus before freezing the native document schema