fix(documentstore): eliminate OOM during vectorstore upsert of large stores

[maxtechera]

Summary

Fixes OOM crash during Pinecone upsert of large document stores (95k+ chunks). Memory climbed steadily from baseline to 2GB over ~1 hour until SIGKILL.

Root Cause

Three compounding memory leaks during vectorstore upsert:

1. addedDocs accumulation (indexing.ts:305) — every batch pushed full Document objects into an array never freed. 95k chunks x ~5KB = ~475MB.
2. indexResult.addedDocs spread (documentstore/index.ts:1841) — each outer batch spread inner results into a growing array. Another ~475MB.
3. totalKeys fetch (indexing.ts:350) — loaded every record manager key into memory just to count them.

Changes

• Cap addedDocs to 4 preview samples instead of accumulating all docs
• Remove addedDocs spread in worker thread (field is omitted from persisted result anyway)
• Replace listKeys({}).length with computed count

Expected Impact

Memory during 95k upsert: ~2GB OOM → ~100-200MB steady-state.

Also scaled Render to Pro Plus (4GB) as belt-and-suspenders.

Related

• KUMELLO-68
• Follow-up to PR fix(AAIDomains): separate Supabase fetch retry from onPage callback #1029 (streaming sync fix)

[linear]

KUMELLO-68 fix(AAIDomains): streaming loader — never hold all documents in memory

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
answerai-docs	Building	Preview	Mar 17, 2026 0:49am
the-answerai	Building	Preview	Mar 17, 2026 0:49am

[claude]

test comment from review bot

[claude]

@/tmp/review.md

[claude]

@/home/runner/work/theanswer/theanswer/.claude/pr_review_1031.md