Repository navigation
Conversation
PDFDocument.load copies every stream's bytes out of the input while parsing, so a large PDF sits in memory twice until the input is freed. With copyStreams: false, ByteStream.slice returns subarray views instead. The default is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
A new
LoadOptions.copyStreams(defaulttrue, so nothing changes unless it's set):With
false, the parser'sByteStream.slicereturnssubarrayviews into the input instead ofslicecopies. Every parsed stream then points into the input bytes rather than holding its own copy.Why?
PDFDocument.loadcopies every stream out of the input while parsing, so a large PDF sits in memory twice until the caller drops the input. When a PDF is split in a browser tab, that copy is the difference between fitting and the tab being killed. We split students' PDFs of up to 200 MB on phones before upload, and phone tabs were dying there.We ship this change as a patch on 2.9.1, together with
PDFRawStream.clonesharing contents (see below). With both, peak memory of our split dropped from +420 MB to +285 MB for a 133 MB scan, and by 16–19% for 53–55 MB PDFs, and the split ran 11–40% faster. In our profiling this change accounts for most of that; the clone change adds roughly one part's size.How?
copyStreamsis threaded fromPDFDocument.loadintoPDFParser.forBytesWithOptionsandByteStream.of.ByteStreamkeeps acopySlicesflag (defaulttrue) and pickssliceorsubarray. Only the top-level parser's stream gets the flag; object streams are parsed from their own decoded buffer and keep copying.The trade-off is in the option's doc comment: the input must not be modified while the document is in use, and it stays in memory as long as any stream points into it. That's why this is opt-in rather than the new default.
I looked for anything that writes into stream contents in place, which would make views unsafe, and found nothing:
updateContentsreplaces the array.Alternative considered: also letting
PDFRawStream.cloneshare contents, which saves another copy per copied page incopyPages. I left it out becausePDFObjectCopier.specasserts that a copy gets its own array. I'm happy to add it behind the same option if you'd like.Testing?
PDFDocument.specload():copyStreams: false;save()produces byte-identical output both ways forv15xref.pdfandnormal.pdf.tsc --noEmit, eslint and prettier are clean.copyPagesand round-tripped withsave(). All 197,352 objects in the outputs were identical.New Dependencies?
No.
Screenshots
N/A. The output is unchanged.
Suggested Reading?
Yes, the stream object sections (7.3.8).
Anything Else?
The option is named after what it controls, not after memory, so it reads naturally next to
capNumbersandignoreEncryption. I'm happy to rename it.Checklist
🤖 Generated with Claude Code