Split oversized Prethink context files into paginated CSVs - #32
Draft
bmuschko wants to merge 1 commit into
Draft
Conversation
bmuschko
force-pushed
the
paginate-oversized-context-files
branch
from
August 31, 2026 18:37
3fc6eff to
05d47d9
Compare
bmuschko
marked this pull request as draft
August 31, 2026 18:38
bmuschko
force-pushed
the
paginate-oversized-context-files
branch
7 times, most recently
from
August 31, 2026 22:50
a9d8677 to
dd23b2e
Compare
bmuschko
force-pushed
the
paginate-oversized-context-files
branch
from
September 1, 2026 15:28
dd23b2e to
47e5928
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Each generated Prethink context table is serialized into a single
PlainTextfile. On a large repository a table such asmethod-quality-metrics.csvcan grow large enough to crash the Moderne CLI serializer, and even a serializer that no longer overflows rejects the file against the reader's per-file size guard. The result is a failed run rather than usable context.Fix
Add a universal guardrail so no single generated context file can grow that large:
<base>.csv; overflow spills into<base>-002.csv,<base>-003.csv, and so on. Consumers read every page with the glob<base>*.csv. Rows are streamed so memory stays bounded.Known limitation
ExportContextis a two-cycle scanning recipe, and the CLI overlay only persists files created in the first cycle, which runs before any row counts are known. The first run of a newly oversized table therefore caps to its single primary file; page capacity then grows geometrically each run the table stays saturated, so it reaches full pagination within a few runs. The guardrail itself holds on every run, an oversized file is never emitted, only the completeness of a brand-new oversized table lags briefly. This is inherent to the two-cycle plus overlay constraint.