Skip to content

Split oversized Prethink context files into paginated CSVs - #32

Draft
bmuschko wants to merge 1 commit into
mainfrom
paginate-oversized-context-files
Draft

Split oversized Prethink context files into paginated CSVs#32
bmuschko wants to merge 1 commit into
mainfrom
paginate-oversized-context-files

Conversation

@bmuschko

@bmuschko bmuschko commented Aug 31, 2026

Copy link
Copy Markdown

Problem

Each generated Prethink context table is serialized into a single PlainText file. On a large repository a table such as method-quality-metrics.csv can grow large enough to crash the Moderne CLI serializer, and even a serializer that no longer overflows rejects the file against the reader's per-file size guard. The result is a failed run rather than usable context.

Fix

Add a universal guardrail so no single generated context file can grow that large:

  • A table whose CSV would exceed a fixed internal budget of 8 MB, comfortably under the CLI serializer's single-string limit, is split at row boundaries. Page one keeps <base>.csv; overflow spills into <base>-002.csv, <base>-003.csv, and so on. Consumers read every page with the glob <base>*.csv. Rows are streamed so memory stays bounded.
  • Page slots are provisioned during the scanning cycle so the CLI edit overlay persists them. A table that outgrows its provisioned pages is capped rather than emitted oversized, and the dropped-row count is surfaced in the generated markdown.
  • The threshold is an internal guardrail rather than a recipe option, since the right value tracks the CLI serializer's limits rather than a user preference. The recipe's public options are unchanged, so existing call sites are unaffected.

Known limitation

ExportContext is a two-cycle scanning recipe, and the CLI overlay only persists files created in the first cycle, which runs before any row counts are known. The first run of a newly oversized table therefore caps to its single primary file; page capacity then grows geometrically each run the table stays saturated, so it reaches full pagination within a few runs. The guardrail itself holds on every run, an oversized file is never emitted, only the completeness of a brand-new oversized table lags briefly. This is inherent to the two-cycle plus overlay constraint.

@github-project-automation github-project-automation Bot moved this to In Progress in OpenRewrite Aug 31, 2026
@bmuschko
bmuschko force-pushed the paginate-oversized-context-files branch from 3fc6eff to 05d47d9 Compare August 31, 2026 18:37
@bmuschko
bmuschko marked this pull request as draft August 31, 2026 18:38
@bmuschko
bmuschko force-pushed the paginate-oversized-context-files branch 7 times, most recently from a9d8677 to dd23b2e Compare August 31, 2026 22:50
@bmuschko
bmuschko force-pushed the paginate-oversized-context-files branch from dd23b2e to 47e5928 Compare September 1, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant