Skip to content

feat: copyStreams load option to read streams as views into the input - #175

Open
anzemur wants to merge 1 commit into
cantoo-scribe:masterfrom
anzemur:load-without-copying-streams
Open

anzemur wants to merge 1 commit into
cantoo-scribe:masterfrom
anzemur:load-without-copying-streams

Conversation

@anzemur

@anzemur anzemur commented Nov 1, 2026

Copy link
Copy Markdown

What?

A new LoadOptions.copyStreams (default true, so nothing changes unless it's set):

const pdfDoc = await PDFDocument.load(bytes, { copyStreams: false });

With false, the parser's ByteStream.slice returns subarray views into the input instead of slice copies. Every parsed stream then points into the input bytes rather than holding its own copy.

Why?

PDFDocument.load copies every stream out of the input while parsing, so a large PDF sits in memory twice until the caller drops the input. When a PDF is split in a browser tab, that copy is the difference between fitting and the tab being killed. We split students' PDFs of up to 200 MB on phones before upload, and phone tabs were dying there.

We ship this change as a patch on 2.9.1, together with PDFRawStream.clone sharing contents (see below). With both, peak memory of our split dropped from +420 MB to +285 MB for a 133 MB scan, and by 16–19% for 53–55 MB PDFs, and the split ran 11–40% faster. In our profiling this change accounts for most of that; the clone change adds roughly one part's size.

How?

copyStreams is threaded from PDFDocument.load into PDFParser.forBytesWithOptions and ByteStream.of. ByteStream keeps a copySlices flag (default true) and picks slice or subarray. Only the top-level parser's stream gets the flag; object streams are parsed from their own decoded buffer and keep copying.

The trade-off is in the option's doc comment: the input must not be modified while the document is in use, and it stays in memory as long as any stream points into it. That's why this is opt-in rather than the new default.

I looked for anything that writes into stream contents in place, which would make views unsafe, and found nothing:

  • updateContents replaces the array.
  • Decryption returns new arrays.
  • The writer only reads.

Alternative considered: also letting PDFRawStream.clone share contents, which saves another copy per copied page in copyPages. I left it out because PDFObjectCopier.spec asserts that a copy gets its own array. I'm happy to add it behind the same option if you'd like.

Testing?

  • New tests in PDFDocument.spec load():
    • streams are copied by default;
    • streams are views into the input with copyStreams: false;
    • save() produces byte-identical output both ways for v15xref.pdf and normal.pdf.
  • Full suite: 907 passed, 1 skipped. tsc --noEmit, eslint and prettier are clean.
  • Our 2.9.1 patch (this change plus clone sharing), before and after on 60 real-world PDFs: split with copyPages and round-tripped with save(). All 197,352 objects in the outputs were identical.

New Dependencies?

No.

Screenshots

N/A. The output is unchanged.

Suggested Reading?

Yes, the stream object sections (7.3.8).

Anything Else?

The option is named after what it controls, not after memory, so it reads naturally next to capNumbers and ignoreEncryption. I'm happy to rename it.

Checklist

  • I read CONTRIBUTING.md.
  • I read MAINTAINERSHIP.md#pull-requests.
  • I added/updated unit tests for my changes.
  • I added/updated integration tests for my changes. (Not needed: default behaviour is unchanged and output is byte-identical.)
  • I ran the integration tests.
  • I tested my changes in Node, Deno, and the browser. (Node/Bun and Chromium; not Deno.)
  • I viewed documents produced with my changes in Adobe Acrobat, Foxit Reader, Firefox, and Chrome. (Output is byte-identical to the default path.)
  • I added/updated doc comments for any new/modified public APIs.
  • My changes work for both new and existing PDF files.
  • I ran the linter on my changes.

🤖 Generated with Claude Code

PDFDocument.load copies every stream's bytes out of the input while
parsing, so a large PDF sits in memory twice until the input is freed.
With copyStreams: false, ByteStream.slice returns subarray views instead.
The default is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant