Part B2 semantic-extraction subagents silently crash past the ~64k output-token turn limit on large/deep-mode chunks; SKILL.md chunking is file-count-based, not output-size-aware

[castflow]

A note on how this report was written: I'm not a professional developer,
and English isn't my first language. I used my AI coding assistant (Claude,
via Claude Code) to investigate this and write up the report below, based on
something we actually hit while using the graphify skill in a real
project. I checked the existing issue tracker first to make sure this
wasn't already reported. Happy to answer follow-up questions, most likely
with the assistant's help.

Summary

When GEMINI_API_KEY/GOOGLE_API_KEY are unset, SKILL.md Part B dispatches
semantic extraction to Claude Code Agent-tool subagents instead of a headless
LLM backend ("the host session itself is the LLM"). Each subagent must
produce its entire chunk's extraction (all nodes/edges/hyperedges for
20-25 files, per Step B1's fixed chunk size) as JSON in a single Write call
within a single conversational turn. On dense corpora, and especially with
--mode deep (richer INFERRED edges → more output per file), a chunk's
output can exceed the ~64,000-output-token ceiling on that single turn.

We directly observed the failure: Claude's response exceeded the 64000 output token maximum. When this happens, the subagent's turn ends without
ever writing graphify-out/.graphify_chunk_NN.json — no partial output, no
error surfaced to the orchestrating skill beyond "the file never appeared."

Why this looked like two different bugs before we found the root cause

From the orchestrator's side, a subagent that hits this ceiling is
indistinguishable from one that simply never responds:

• If you don't wait long enough, it looks like an infinite hang.
• If you wait, chunks that stay just under the ceiling still take a long
  time to generate tens of thousands of tokens of JSON (12-28 minutes in our
  runs), so it looks like extreme, unexplained per-chunk latency with no
  correlation to chunk file-count.
• Only once a chunk actually exceeds the ceiling does the real error surface
  — and per Step B3, a missing chunk file is currently diagnosed as "the
  subagent was likely dispatched as read-only (Explore type)", which is a
  plausible-but-wrong explanation for this failure mode: the subagent type
  was correct, it simply crashed mid-generation.

Root cause

Step B1 sizes chunks purely by input file count (20-25 files, hardcoded),
with no accounting for expected output volume — which varies a lot by
mode (--mode deep extracts materially more INFERRED edges per file) and by
document density. There's no equivalent, for this dispatch path, of the
adaptive-retry / bisection logic that the headless backends have for
finish_reason == "length" (see #730, #1372 for the claude/openai
backends' hardcoded-max_tokens truncation handling, and #1611/#1063 for the
claude-cli subprocess backend's hollow-response/bisection recovery) —
because this path never goes through graphify/llm.py at all. It's Claude
Code's own Task-tool subagent mechanism standing in as the LLM, and nothing
in SKILL.md estimates whether a given chunk's expected output fits in one
turn before dispatching it.

Impact

• Silent, total loss of a chunk's extraction (not just truncation — the file
  never gets written, so none of that chunk's files get cached this run).
• The wrong diagnostic in Step B3 sends users chasing "did I use the right
  subagent_type" instead of "this chunk was too big."
• Disproportionately hits exactly the corpora graphify's docs pitch hardest
  for --mode deep (dense, cross-referenced, entity-rich documents) — deep
  mode is the trigger, not an edge case.

Suggested fix

1. Size Step B1's chunks by estimated output, not file count. A rough
   heuristic (bytes-per-file × a mode-dependent multiplier, capped to a
   token budget with margin under 64k) would catch most cases before
   dispatch. The extraction-spec prompt could carry the same budget as
   guidance to the subagent (e.g. "~60 nodes / ~80 edges per chunk, prioritize
   the most cross-referenced entities") so the model self-limits even when
   the pre-dispatch estimate is off.
2. Give Step B3 a second diagnosis branch for missing chunk files.
   Alongside the existing "likely read-only dispatch" guess, surface
   "chunk may have exceeded the model's output-token limit — retry with a
   smaller chunk" when a chunk is missing and its file count/estimated
   density is high. (If the Agent-tool result exposes a stop/error reason to
   the orchestrating skill, prefer branching on that directly over guessing
   from symptoms.)
3. Consider incremental writes for large chunks — have the subagent
   append nodes in batches across multiple Write/Edit calls instead of
   one terminal Write, so a chunk that would exceed the ceiling degrades to
   "wrote N of M nodes" rather than losing the whole chunk. More invasive,
   but it turns a hard failure into a partial one.

Relation to existing issues

• Hardcoded max_tokens=8192 causes truncation cascade on dense docs (3× cost overhead) #730 / OpenAI backend caps output at 8192 tokens (half the 16384 other backends use) → truncated semantic chunks on dense repos #1372: hardcoded max_tokens truncation in the headless
  claude/openai API backends (graphify/llm.py). Different code path —
  those backends have adaptive retry that catches this class of failure;
  our path (Agent-tool subagent dispatch, used when no API key is set) has
  no such mechanism at all.
• fix(claude-cli): deliver extraction instructions in the user turn (fixes infinite bisection on Claude Code >= 2.1) #1611 / Fix hollow-response loop in claude-cli backend (re-targeted to v8, clean diff, with tests) #1063: response-parsing and prompt-delivery fixes for the
  claude-cli subprocess backend (_call_claude_cli shelling out to the
  claude binary). Also a different mechanism from Claude-Code-native
  Task-tool subagents — no subprocess is involved in Part B2's dispatch.
• [RFC] Native graphify update for the semantic layer — a scripted conductor with a pluggable LLM hook (we measured: the math is 0.89s–52s, the agent-walked workaround is 11–14 min) #1710: RFC observing that "agent-walked" execution of the update
  conductor is slow (11-14 min) compared to a scripted equivalent (under a
  minute). Consistent with what we saw, and a good argument for a scripted
  path in general — but that RFC is about eliminating agent-walked execution
  for the update conductor steps (detect/merge/cluster/label), not about
  this specific extraction-subagent output-ceiling crash, which would still
  need addressing even inside a scripted conductor if it still shells out to
  an agent for the actual semantic-extraction step (as the RFC's own
  proposal, step 3, does: "a pluggable LLM hook... claude -p/any agent CLI
  supplied by the user").

Environment

• graphifyy 0.8.36
• Semantic extraction via SKILL.md Part B2, Claude Code Agent-tool
  subagents, subagent_type="general-purpose", model="sonnet"
• --mode deep
• Windows 11
• Directly observed error: Claude's response exceeded the 64000 output token maximum (one occurrence, during a 13-file incremental chunk)

Our workaround (not a library fix)

For the run where we hit this, we sidestepped re-extraction entirely: most
of the "uncached" files had only changed by line-ending normalization (CRLF),
so we re-keyed their existing cache entries under the new content hash
instead of re-extracting, and folded the small amount of genuinely new
content into hand-written inline fragments rather than a subagent dispatch.
Going forward we're manually holding chunk subagents to a "~60 nodes / ~80
edges" budget via prompt guidance and sizing chunks by estimated output
rather than file count — a prompt-level mitigation on our side, not a fix to
SKILL.md itself.