Skip to content

diffr web: pack grammar tables in the browser engine (127 MB → 25 MB) - #1095

Open
milanb17 wants to merge 1 commit into
claude/diffr-web-mobile-rendererfrom
claude/diffr-web-packed-grammars
Open

milanb17 wants to merge 1 commit into
claude/diffr-web-mobile-rendererfrom
claude/diffr-web-packed-grammars

Conversation

@milanb17

Copy link
Copy Markdown
Contributor

Summary

Pack the browser engine's grammar parse tables and unpack each language the first time a file in it is diffed. Stacks on #1094; merge it first.

before: 126.6 MB engine = 14.2 MB code + 111.0 MB grammar tables
after:   25.2 MB engine = 14.3 MB code +   9.6 MB packed tables
  • Native builds already packed the tables with zstd. The build script now does the same for wasm, finding each table in the object file's data segments and linking symbol table with wasmparser.
  • In the browser, ruzstd (pure Rust) provides the ZSTD_decompress the unpacker calls.
  • The gzipped download goes from 11.7 MB to 8.7 MB.

Evidence

  • Old and new engines produced identical output for 57 of 57 inputs: 49 files from this repository, plus samples for the largest grammars it has no files in (F#, Julia, Verilog, LaTeX, Fortran, Objective-C, SQL, VHDL).
  • iPhone 17 Pro simulator, vite v5.3.0...v5.4.0 (217 files), cold loads, three runs: peak 507–537 MB and settled 280–283 MB, down from 696–759 MB and 374–422 MB with the previous PR alone.
  • diffr-grammars passes rustfmt and clippy for native and wasm32. The crate has no tests; the native path is unchanged.

Merge Danger

Door: two-way; revert the commit and rebuild the engine.

Blast Radius: the browser engine's grammars. A table that unpacked wrongly would break parsing for that language only; the comparison above found none. ZSTD_decompress now appears among the engine's exports in the generated bindings.

The browser engine carried every grammar's parse tables uncompressed,
111 MB of its 127 MB. The wasm build now packs them the way native builds
do, with ruzstd standing in for libzstd, so the engine is 25 MB and only
the languages a comparison uses are unpacked.

Agent-Session: 429958d8-68f3-4f68-bd3b-a8371e7feb8e
Agent-Session: 5eb4067a-b2f6-4508-a6dd-0471ff2c14d1
Agent-Session: c4502e33-93da-4505-ac5c-2015d3e90139

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant