Skip to content

fix(index): exclude large trees and keep dialogue record off the full graph - #59

Merged
Roarpeng merged 3 commits into
mainfrom
cursor/exclude-globs-and-dialogue-read-d8ac
Oct 7, 2026
Merged

Roarpeng merged 3 commits into
mainfrom
cursor/exclude-globs-and-dialogue-read-d8ac

Conversation

@Roarpeng

@Roarpeng Roarpeng commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Why

Issue #56: a 61k-file workspace produced a 1.2GB file graph. Every user message spawned dialogue record, which read that entire document into the JS heap (2–3.7GB) and OOM'd the host. The reporter's .graphflowignore vendor patch cut the graph to ~9MB. Upstream had no supported way to exclude directories, and the dialogue path always materialized the store.

Markdown is already capped at DEFAULT_MAX_FILE_SIZE (200KB). DEFAULT_DOCUMENT_MAX_FILE_SIZE (5MB) applies to office/PDF conversion only, so that constant was not lowered.

Changes

  • graphPolicy.excludeGlobs and graphPolicy.maxFileSizeBytes flow through workspace index, single-file index, freshness checks, graph index / run / selfcheck.
  • <workspace>/.graphflowignore uses the same rules as the attached patch: name/ at any depth, anchored path/to/dir/, basename globs, path globs (* does not cross /, a double-star does), inline # comments. Negation and nested ignore files are not supported. Backslashes in rules are normalized to / before matching, and walked paths are compared as POSIX relatives, so the same file works on Windows.
  • File stores larger than 512MB: dialogue record streams only dialogue nodes and appends a delta line. It does not JSON.parse the code graph. A later full read folds the delta. SQLite listDialogueNodes() selects dialogue rows in SQL instead of readSnapshot().

Validation

Linux:

  • npx tsc -p tsconfig.json --noEmit — clean
  • npx vitest run tests/m-workspace-exclude-and-dialogue-read.test.ts tests/m83-index-scan-budget.test.ts tests/m11-file-indexer.test.ts tests/dialogue-thread.test.ts tests/m-dialogue-temporal.test.ts tests/graphify-file-client-cache.test.ts tests/m24-sqlite-backend.test.ts tests/m-dialogue-search-recall.test.ts — pass

The huge-store test lowers the materialization threshold to 1 byte, records two dialogue turns, and asserts the base JSON is not parsed (getGraphifyFileStoreParseCount() === 0), the base file still holds only the filler node, and the delta log holds the turns. After the threshold is restored, readSnapshot() returns both the filler node and the dialogue nodes. SQLite asserts listDialogueNodes() does not call readSnapshot().

Windows path joining is covered by the ignore-rule unit test (tools\\cache/ matches tools/cache/a.ts) and by normalizePath on walker relatives. This agent cannot run windows-latest; that CI job is the platform check.

Fixes #56

Open in Web Open in Cursor 

… them

Huge workspaces were indexed in full, then every dialogue record read the
whole file graph into the JS heap. graphPolicy.excludeGlobs, maxFileSizeBytes,
and .graphflowignore narrow the walk. On a file store over 512MB, dialogue
record streams only dialogue nodes and appends a delta; sqlite lists those
rows with SQL instead of readSnapshot.

Fixes #56
@Roarpeng
Roarpeng marked this pull request as ready for review October 7, 2026 09:42
cursoragent and others added 2 commits October 7, 2026 09:45
…e fix

CHANGELOG conflicted after #57. Both Unreleased notes stay: the MCP SDK
and proxy-addr override, and excludeGlobs / streaming dialogue record.

Co-authored-by: RoarPeng <Roarpeng@users.noreply.github.com>
… fix

CHANGELOG conflicted after #58. Both Unreleased notes stay.
@Roarpeng
Roarpeng merged commit c90240e into main Oct 7, 2026
6 checks passed
@Roarpeng
Roarpeng deleted the cursor/exclude-globs-and-dialogue-read-d8ac branch October 8, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 大体量工作区把宿主拖到 OOM:1.2 GB 图谱被每条消息整读;附 .graphflowignore 补丁、实测数字与上游建议

2 participants