Repository navigation
fix(index): exclude large trees and keep dialogue record off the full graph - #59
Merged
Merged
Conversation
… them Huge workspaces were indexed in full, then every dialogue record read the whole file graph into the JS heap. graphPolicy.excludeGlobs, maxFileSizeBytes, and .graphflowignore narrow the walk. On a file store over 512MB, dialogue record streams only dialogue nodes and appends a delta; sqlite lists those rows with SQL instead of readSnapshot. Fixes #56
Roarpeng
marked this pull request as ready for review
October 7, 2026 09:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Issue #56: a 61k-file workspace produced a 1.2GB file graph. Every user message spawned
dialogue record, which read that entire document into the JS heap (2–3.7GB) and OOM'd the host. The reporter's.graphflowignorevendor patch cut the graph to ~9MB. Upstream had no supported way to exclude directories, and the dialogue path always materialized the store.Markdown is already capped at
DEFAULT_MAX_FILE_SIZE(200KB).DEFAULT_DOCUMENT_MAX_FILE_SIZE(5MB) applies to office/PDF conversion only, so that constant was not lowered.Changes
graphPolicy.excludeGlobsandgraphPolicy.maxFileSizeBytesflow through workspace index, single-file index, freshness checks,graph index/ run / selfcheck.<workspace>/.graphflowignoreuses the same rules as the attached patch:name/at any depth, anchoredpath/to/dir/, basename globs, path globs (*does not cross/, a double-star does), inline#comments. Negation and nested ignore files are not supported. Backslashes in rules are normalized to/before matching, and walked paths are compared as POSIX relatives, so the same file works on Windows.dialogue recordstreams only dialogue nodes and appends a delta line. It does notJSON.parsethe code graph. A later full read folds the delta. SQLitelistDialogueNodes()selects dialogue rows in SQL instead ofreadSnapshot().Validation
Linux:
npx tsc -p tsconfig.json --noEmit— cleannpx vitest run tests/m-workspace-exclude-and-dialogue-read.test.ts tests/m83-index-scan-budget.test.ts tests/m11-file-indexer.test.ts tests/dialogue-thread.test.ts tests/m-dialogue-temporal.test.ts tests/graphify-file-client-cache.test.ts tests/m24-sqlite-backend.test.ts tests/m-dialogue-search-recall.test.ts— passThe huge-store test lowers the materialization threshold to 1 byte, records two dialogue turns, and asserts the base JSON is not parsed (
getGraphifyFileStoreParseCount() === 0), the base file still holds only the filler node, and the delta log holds the turns. After the threshold is restored,readSnapshot()returns both the filler node and the dialogue nodes. SQLite assertslistDialogueNodes()does not callreadSnapshot().Windows path joining is covered by the ignore-rule unit test (
tools\\cache/matchestools/cache/a.ts) and bynormalizePathon walker relatives. This agent cannot runwindows-latest; that CI job is the platform check.Fixes #56