Merge upstream main (c6aaa20) into fork main - #6
Merged
Conversation
…ables siblings (colbymchenry#1351) (colbymchenry#1370) The Codex installer's `findNextTableHeader` skipped `[[array-of-tables]]` headers instead of treating them as a block boundary, so any `[[...]]` block after `[mcp_servers.codegraph]` in ~/.codex/config.toml was silently deleted on install/upgrade/uninstall. Now treats both `[...]` and `[[...]]` as boundaries, with a small line lexer so header-shaped text inside multiline strings/arrays isn't mistaken for a boundary. Adds round-trip regression coverage (install → reinstall → uninstall) + CHANGELOG entry. Fixes colbymchenry#1351. Supersedes colbymchenry#624. Thanks @KtzeAbyss.
…ains
Adds three new installer targets so `codegraph install` can wire the
MCP server into GitHub Copilot surfaces:
- copilot-vscode: .vscode/mcp.json (local) or the VS Code User-dir
mcp.json (global), JSONC-surgical edits, `--path` pinned via
${workspaceFolder} for global installs
- copilot-cli: ~/.copilot/mcp-config.json
- copilot-jetbrains: github-copilot config dir (XDG / %LOCALAPPDATA%)
Detection, install, uninstall, and --print-config are covered for all
three in installer-targets.test.ts, including platform-specific path
resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
….copilot/ide locks
The VS Code Copilot Chat extension writes MCP socket-handoff lock files
into ~/.copilot/ide/ on launch, so `existsSync(~/.copilot)` reported the
Copilot CLI as installed on any machine that merely has the VS Code
extension (caught live on the maintainer's Mac). Detection now counts
the dir as a CLI footprint only when it holds something besides `ide`.
Also: uninstalling a from-scratch install now deletes mcp-config.json
instead of leaving a `{}` husk that would keep detect() reporting the
CLI as installed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… folder
VS Code refuses to start a user-level MCP server whose entry uses
${workspaceFolder} in a window with no folder open, surfacing only a
cryptic "Variable workspaceFolder can not be resolved" toast (hit live
during validation). Global installs now note this up front.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…— VS Code toasts an error in every folderless window
A user-level mcp.json entry using ${workspaceFolder} makes VS Code
refuse to start the server in ANY window without a folder open (loose
files, welcome tab), toasting "Variable workspaceFolder can not be
resolved" — recurring error-noise, hit live during validation.
The pin was never needed for VS Code: unlike Cursor, VS Code documents
stdio-server cwd as the workspace folder, and the codegraph server
resolves its project via roots/list with a cwd fallback. Global entries
are now variable-free (`serve --mcp`); local installs keep the absolute
--path. This supersedes the "open a folder" install note from the
previous commit, which is removed again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ows (colbymchenry#1466) (colbymchenry#1489) The standalone bundle's bin dir exposes only codegraph.cmd, and Claude Code executes UserPromptSubmit hooks through Git Bash, which applies no PATHEXT — so the bare `codegraph prompt-hook` the installer wrote was "command not found" (exit 127) on every prompt. Write the platform-correct spelling, recognize both spellings on uninstall/opt-out, and self-heal an installer-written entry from the other platform in place on install/upgrade re-runs (npx/hand-edited variants stay untouched). Reproduced and validated on the Windows VM: bare form exits 127 under Git Bash on a standalone-only PATH, codegraph.cmd exits 0; full installer suite (165 tests, including the new migration coverage) green on Windows + macOS. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…thout bound (colbymchenry#1431) (colbymchenry#1490) A SIGKILL'd process (the colbymchenry#850 liveness watchdog, OOM, a crash) leaves its WAL on disk; the next session appends to the same file; and nothing ever truncated it — PASSIVE checkpoints fold frames but keep the file at its high-water mark, and the one shrinking path (a clean last-connection close) is exactly what a killed-daemon world never takes. Observed at 25.6 GB on a 5.46 GB DB, growing until the disk filled. - journal_size_limit on every connection: resetting checkpoints now clip the WAL back to the cap instead of leaving it at its high-water mark. - healOversizedWal() fired from every DatabaseConnection.open: off-thread PASSIVE fold + TRUNCATE when the leftover WAL exceeds the cap (64 MB, CODEGRAPH_WAL_HEAL_MB to override). Single-flight per connection with bounded retries — concurrent passes defeat each other (each checkpoint sees the other as a busy reader). - Daemon/direct MCP watchdogs now pass progressPaths (DB + WAL), extending the colbymchenry#1231 slow-disk deferral to the long-lived server so a healthy daemon mid slow statement isn't SIGKILL'd — fewer kills, fewer leaked WALs. - codegraph status shows WAL size (human + JSON) and warns when it dwarfs the DB; daemon.log lines and the watchdog kill notice now carry ISO timestamps so kills can be placed in time. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ed from its index (colbymchenry#1474) (colbymchenry#1492) codegraph_node / codegraph_explore read CURRENT bytes but slice them at INDEXED line ranges; after an un-synced edit that slice can be a DIFFERENT symbol's code served under the requested name — isError: false, introduced by the 'verbatim … do not Read' guarantee. The watcher-based pending (colbymchenry#403) and degraded (colbymchenry#876) banners cannot cover a project reached via projectPath: cross-project instances have no watcher, by construction. Freshness is now verified at the point of emission from data the index already stores: one stat per rendered file (size + floored mtime, the sync fast path's own test), sha256 content-hash compare only on stat mismatch (so a touch/identical rewrite never false-positives), memoized briefly per handler. On drift: - codegraph_node: small files ship WHOLE and CURRENT (Read-parity, still no Read needed); large ones omit the body with an explicit notice steering to the tool's file-read mode or Read. Location/signature stay, flagged as possibly shifted. - codegraph_explore: the whole-file render (already correct by construction) is kept and flagged; adaptive/skeleton/cluster slicing is disabled for drifted files — a too-big drifted file is omitted with a notice instead. The verbatim/do-not-Read header gains a per-file exception, and a trailing note flags shifted line references (flow, blast radius, symbol lists). The guarantee itself is preserved: everything actually rendered is still byte-accurate — drifted files ship whole or not at all, never as a possibly-wrong slice. A re-sync of the target project restores normal output (covered by test). Adds __setLoadCodeGraphForTests (same seam pattern as __setFsWatchForTests) so in-process tests can exercise a genuine cross-project open, which vitest's transform cannot service through the lazy require. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
colbymchenry#1478) (colbymchenry#1493) Python's class-as-value idioms (return SomeClass, x = SomeClass, registry dicts, classes passed as arguments) produced no references edges, so callers/impact on a Django/DRF serializer missed the views that consume it. Three gates dropped them: - return_statement was never dispatched by PYTHON_SPEC (kernel mirrored) - the extraction gate (definedHere) collected function/method names only - resolution accepted function/method targets only (matchFunctionRef + the function_ref import fast path) Capture return_statement for Python (single expression; tuple returns not descended), admit same-file CLASS names to the gate, and accept class targets for Python bare identifiers — scoped to Python so the TS/JS KIND FILTER contract is untouched. The docopt false-positive mechanism behind the function-only rule (lowercase locals vs same-named methods) doesn't transfer: methods stay excluded for bare ids, and the same-file/import gate + unique-or-drop rules still apply. Probed on django-rest-framework (~250 files): 559 new references→class edges, 10/10 sampled genuine (serializer_class = AuthTokenSerializer, the ModelSerializer field-mapping registry, aliases, ctor args, isinstance). EXTRACTION_VERSION 24 → 25. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…coverage (colbymchenry#1475) (colbymchenry#1494) The "no covering tests found" flag only inspected a symbol's direct callers, so helpers exercised transitively by tests (logDebug runs 1,471x under npm test) were reported untested — wrong for ~40% of flagged symbols per the issue's measurement. The check now BFSes up the caller graph (3 hops, 64-lookup budget per entry) and reports indirect coverage as "tested via callers: <files>". When nothing is found it claims only what was measured — "no tests found within 3 caller hops", or the weaker "no test calls this directly" if the budget ran out — and drops the warning glyph. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rd (CG-7) (colbymchenry#1497) * feat(telemetry): D1 schema + migrations for raw events and daily rollups First step of replacing PostHog with self-hosted telemetry on Cloudflare D1. Creates the codegraph-telemetry database binding and the initial migration; no worker code paths change yet (the ingest write path and the nightly rollup cron land next). Schema is raw events plus daily rollups: `events` holds one row per sanitized event with the envelope broken out into columns and event-specific props as JSON; `daily_machines`, `daily_event_counts` and `daily_dim_counts` are the nightly rollups the dashboard reads; `machine_first_seen` and `machine_days` carry the retention cohorts and are never purged. One generic dimension table covers every bar and pie, so a new breakdown is a cron change rather than a migration. The migration is commented as an audit surface, like the rest of this worker — every column, and which dashboard chart each rollup table serves. Three judgment calls worth flagging, all documented in the file: - `events` gets `(day, event)` instead of the separate `(day)` and `(event, day)` indexes. D1 bills a row write per index touched, so a third index on the hot table costs ~97k writes/day, and `(day, event)` is a covering index for plain day-range scans anyway (verified with EXPLAIN QUERY PLAN). - `daily_event_counts` and `daily_dim_counts` carry a `machines` column, and `machine_days` a `prod` flag. The "users by ..." panels and the production-user count are distinct-machine numbers, not event counts, and they are unrecoverable once raw events are purged. - No CHECK constraint on `event`: the worker's allowlist is the source of truth and the write path is fail-silent, so a rejected INSERT would lose data quietly instead of erroring loudly. Volume note in the migration footer: ~30M row writes/month against the 50M included on Workers Paid. Storage is the tighter constraint — raw events grow ~74 MB/day, so retention should start at 90 days (~6.7 GB) rather than 180, which would exceed D1's 10 GB per-database cap. * feat(telemetry): admin dashboard worker — scaffold + shared-password auth New Cloudflare Worker at telemetry-dashboard/, sibling of telemetry-worker/ and bound read-only to the same D1 database. Serves a static frontend plus a JSON API behind a shared password, on stats.getcodegraph.com. Auth is the simplest thing that is actually safe for exactly two users: one password in a secret, compared in constant time over SHA-256 digests, and an HMAC-signed cookie (HttpOnly; Secure; SameSite=Lax; Path=/) with a one-year expiry so you sign in once per browser. The cookie is a signed assertion, not a lookup key — no session store. Its payload carries a fingerprint of the password it was minted against, so rotating ADMIN_PASSWORD signs everyone out. Login attempts are capped at 5/min per IP via a ratelimit binding. Everything is deny-by-default: assets.run_worker_first routes every request through the worker before the static-asset server sees it, so the dashboard HTML, its JS, its CSS and the chart library are all behind the session check. The login page is rendered inline by the worker rather than served from public/, which leaves no "is this file public?" judgement calls in the asset directory. Unauthenticated pages 302 to /login, unauthenticated /api/* gets 401. A missing secret fails closed rather than opening the dashboard. scripts/smoke-auth.sh is the regression net — 54 assertions against a throwaway `wrangler dev` covering the gate, cookie flags and persistence, forged/flipped/ truncated cookies, open-redirect refusal, brute-force capping, and password rotation invalidating live sessions. Refs CG-11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(telemetry-dashboard): simplify the chart-library probe in the shell Refs CG-11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(telemetry): nightly rollup cron + raw-event retention purge (CG-10) Adds a scheduled() handler to the ingest worker that recomputes daily_event_counts / daily_dim_counts / daily_machines for the just-completed UTC day plus a 2-day overlap (late-arriving offline buffers), then purges raw events past the retention window. Rollup writes are idempotent upserts, so a re-run never double-counts. Also adds an ADMIN_TOKEN-guarded POST /admin/rollup?day=YYYY-MM-DD for backfill/repair, and drops the PostHog forwarding path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(telemetry): dashboard charts — SQL API over D1 + the Chart.js views (CG-12, CG-13) Replaces the scaffold page with the dashboard proper: 19 panels covering every view of the PostHog dashboard this retires, driven by one filter row. src/api.ts is the read API CG-12 specified: /api/{meta,summary,timeseries, breakdown,activation,retention}, all range-scoped, all parameterized against a closed set of dims and metrics, all shaped labels[] + datasets[] so the frontend does no arithmetic. Rollups answer everything except the activation funnel, which needs raw events and says where they start. The frontend splits into a DOM-free panel registry (public/panels.js) and the page that mounts it (public/app.js), so the render check can drive the same registry the browser rendered from. Panels fail alone, refetch dims rather than flashing, and every chart carries a table twin. Two numbers are labelled rather than rounded off: range-wide "users" per dimension is machine-days (the rollups cannot give distinct machines, and per-day counts are taken as the largest single-event count so one machine's install + index + usage is not counted three times), and recent activation and retention cohorts are marked as still-converting instead of drawn as a cliff. Both colour scales were run through the data-viz validator against the panel surface, not picked by eye; the results are recorded in public/theme.js. Verification, all against the committed fixture (12 machines over 10 days, every expected number worked out by hand from the events, not recorded from a run): scripts/smoke-api.sh 98 assertions scripts/render-check.mjs 79 assertions — real Chromium over CDP, no new deps scripts/smoke-auth.sh 54 assertions (unchanged, still green) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(telemetry): cutover runbook + the end-to-end gate that de-risks it (CG-14) The account-level steps of the PostHog cutover are the maintainer's to run, so this lands the runbook they follow and the check that has to pass first. The runbook (telemetry-worker/README.md) walks the six steps in the order that keeps them reversible: Workers Paid → migrate → deploy → watch 24h → verify the first rollup and the dashboard → only then delete POSTHOG_KEY and cancel the subscription. Step 3 records the outgoing version id because `wrangler rollback` is the escape hatch for the whole verification window, and that window is precisely why the PostHog key is deleted last rather than first. The new gate (scripts/smoke-cutover.sh, `npm run smoke:cutover`) covers the one seam nothing else did. Both workers declare the same D1 database_id, so pointing them at a single --persist-to directory runs the real chain: a client batch → the ingest worker → D1 → the nightly rollup → the dashboard API reading the numbers back. Every other suite stops at one link — smoke-ingest at the events table, smoke-rollup at hand-checked SQL, smoke-api at a hand-written fixture that the cron never touched. That left the dimension names the rollup WRITES versus the ones the dashboard READS agreeing by convention across two branches, where a mismatch is silent: no error, no failed request, just a panel reading zero forever. 61 assertions, all 13 dimensions, and three deliberate traps — a ci machine that is active but not a production user, usage_rollup counts that must be summed rather than tallied, and an uninstall's `targets` that must not leak into the install-scoped breakdown. Writing it caught that the activation funnel's denominator is first-seen machines, not install events (deliberate — a reinstall must not re-enter the funnel), so the suite now pins that distinction rather than assuming it. Also rewords the last PostHog reference in dashboard code: a comment justifying the 14-day retention curve by pointing at a dashboard step 6 deletes. The reasoning now stands on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(telemetry): tell the truth about where events are stored (CG-15) The telemetry docs are a privacy contract, and they still described a managed analytics store that no longer receives anything. Replace that with what actually happens now — events land in our own D1 database on Cloudflare, the endpoint makes no outbound requests, raw events are purged after 90 days and only anonymous daily rollups outlive them. This strengthens the guarantee rather than restating it: there is no second party to share with. - TELEMETRY.md: new "Where it is stored" section; the never-collected IP bullet no longer leans on a vendor-side setting to hold. - docs/design/telemetry.md: ingest section rewritten around D1 + the nightly rollup/retention cron; volume math redone on Workers Paid and the D1 quota (storage, not writes, is what sets the 90-day window); new section documenting the dashboard worker and cross-linking it. - Fixed three drifts from the worker allowlist the sweep surfaced: schema_version was still 1, client_name/client_version was still marked "plumbing to add" though session.ts passes it today, and the legacy sqlite_backend field the worker still accepts was undocumented. - telemetry-worker/README.md: step 6 claimed a repo-wide grep came back clean, which this runbook itself falsifies. Added step 7 — deleting the runbook is what makes that grep true, and is the completion check. - smoke-cutover.sh: the vendor guarantee is now asserted by class (no analytics-ingest endpoint referenced) rather than by one vendor's name, so it keeps working once the name is gone. Verified it still catches a planted forwarding URL. 61/61 pass. Retention is documented as 90 days, not the 180 in the task notes: 180 days of raw events exceeds D1's 10 GB per-database cap, and the code purges at 90. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: untrack local Kommandr issue DB and ignore its sqlite artifacts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…eferrer policy locked Chromium out (CG-16) The dashboard sends Referrer-Policy: no-referrer, and Chromium's behavior on a same-origin form submit from such a page is to send Origin: null. isSameOriginPost() fed "null" to new URL(), which throws → false → 400 "bad request" for every Chromium user typing the correct password. Treat a null Origin like an absent one: it is an unattributed origin, not a foreign one — curl (no Origin at all) was always allowed, the login POST carries no session to ride, and the password is the credential. Real foreign origins stay rejected. The regression net now posts the way Chromium actually does: the smoke-auth sign-in and logout carry Origin: null, and cross-origin logout gets its own rejection case (54 → 56 assertions). The suites missed this because every passing login came from curl or Node fetch — neither sends an Origin header — while render-check injects its cookie past the form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
How codegraph_explore divides its byte envelope among files was unobservable — you could read a response and guess, but not say "this file took 16% and that one took 20%." Nothing else in the budget- allocation epic is measurable without that. CODEGRAPH_EXPLORE_DEBUG now emits one report per explore call (stderr table, stderr JSON, or a JSONL sidecar path). Per file: relevance score, graph mass, term hits, ranking flags, render mode, bytes allocated vs delivered, both shares, and whether it was clipped — plus why a ranked candidate never rendered. Totals cover envelope vs maxOutputChars vs the hard ceiling, the source/meta split, the selection funnel, and the score floor and relevance-gate thresholds applied. Allocated and delivered are reported separately on purpose: they diverge exactly when the 25K ceiling truncates, and conflating them is how a dropped trailing file goes unnoticed. Off by default and byte-identical when off — it ships in the product binary, and a diagnostic that perturbs the response by one byte would invalidate every A/B taken with it on. ExploreDiagnostics.start() returns null unless the env var is set, so every call site is a `diag?.` no-op. Baseline recorded in docs/design/explore-budget-allocation.md: on this repo, src/mcp/tools.ts gets 15.8% of the envelope while three weakly- relevant agent-eval scripts take 61% between them — despite tools.ts carrying 5.4x the score and 2.6x the graph mass of any of them. Small files ship whole; the large answer file is clipped at maxCharsPerFile. Rank ordering is correct and buys nothing. The loop also allocated 23,193 chars against an 18,000 budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mchenry#1500) `isGeneratedFile` was path-only, but Go's own convention is a CONTENT marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one. A Go monorepo with generated CRUD in ordinarily-named files sitting beside hand-written use-cases was therefore invisible to every generated-file down-rank in the codebase — that is colbymchenry#1500. Measured on kubernetes/client-go (2,453 Go files): the canonical banner appears in 2,001 of them, the path check flags 0, the new content check flags exactly those 2,001 — no false positives, no misses. Design: decide at INDEX time (content is already in memory for parsing), persist on `files.generated`, read from the DB. Explore never reads file headers per request. - `hasGeneratedHeader(content)` recognizes the standard banners — Go's, protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator, FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an 8KB/60-line header window, a comment-line requirement (leader or open block comment), and markers tight enough that prose can't trip them. A generator's own source, holding the banner as a string constant in its body, is not flagged; neither is this module itself (pinned by test). - `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback. - Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no backfill: the flag derives from content the migration cannot see, so rows stay 0 until a re-index and every reader unions the flag with the path check — an un-migrated index keeps pre-colbymchenry#1500 behavior rather than regressing. Re-index required; noted in the CHANGELOG. - `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1) lookups. Bounded, not cached: no invalidation, so a ranking call can never serve a verdict the last sync already replaced. Wired into explore ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the context formatter, and the dominant-file/route-file hygiene filters. Cost (acceptance bar was no measurable index-time regression): a single unanchored `/generat/i` test over the header rejects ~every hand-written file before any line splitting. 4.6 µs/file on client-go (worst case — 82% generated). End-to-end `codegraph init` on client-go, n=3 alternating arms: 5.73s median with detection vs 5.76s path-only baseline; the arms cross over between runs, so the difference is inside run-to-run noise. Scope note: generated status remains a stable TIEBREAK at equal score, exactly where it was. Making it a strong negative signal is CG-10, which this unblocks by making the signal correct and available. Two pre-existing tests hard-coded schema version 8; both now track CURRENT_SCHEMA_VERSION (or the migration table) so future migrations don't require editing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ation (CG-6) Two permanent fixtures pinning the failure mode from issue colbymchenry#1500 — explore spending its byte envelope on files that merely name-collide with the query. BOTH FAIL TODAY, by design: they document the bug and become the pass gate for CG-10 (scoring) + CG-12 (proportional allocation). __tests__/fixtures/payroll-go/ — a synthetic Go service mirroring the reporter's shape: generated FKIT CRUD beside a hand-written payroll use-case, entered from an HTTP route. Half the generated tree carries ORDINARY names detectable only by their `// Code generated ... DO NOT EDIT.` header (the colbymchenry#1500 case, and end-to-end cover for CG-5); `payrollpb/*.pb.go` covers the path-detectable channel. BuildPayslip, Upsert and Store each exist twice, generated and hand-written. cycle.go sits above the whole-file window so it clips; the generated files sit below it so they ship whole. Asking "how does payroll cycle create and calculate payslips?" — naming none of the answering symbols — the generated CRUD delivers 57.4% of the envelope against the hand-written layer's 25.6%, all of the latter domain types. cycle.go is allocated the single largest slice (30.6%) and delivers ZERO: the hard ceiling drops its whole section. runPayrollCycleAll, the hand-written BuildPayslip and the real Upsert never reach the agent. The second fixture is this repo, "how does explore allocate its output budget across files", where scripts/agent-eval/*.mjs take 71.8% against tools.ts's 18.5% despite scoring 4.6x lower. It reads the live index, so its assertions are relative rather than fixed percentages. - scripts/agent-eval/probe-allocation.mjs — per-file budget-share probe, driving the CG-4 diagnostic through a JSONL sidecar so it measures the shipping allocator. Fixture entries are hermetic (copy + re-index per run, verified byte-identical across runs); exits 1 while any assertion fails. - scripts/agent-eval/allocation-fixtures.json — both fixtures declared, with the 2026-08-03 baselines. - __tests__/explore-allocation-1500.test.ts — fixture-shape assertions green today; the allocation assertions held as `it.fails` so the suite stays green while the bug is open and goes RED the moment it is fixed. Also documented and deliberately left unfixed: runPayrollCycleAll's `s.store.Upsert` edge resolves to the GENERATED Store.Upsert, not the hand-written one — same-name method resolution across two packages picks the wrong receiver. It is upstream of the allocation bug, so it belongs with CG-10's scoring work. Refs colbymchenry#1500 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ame-collision matches (CG-10, colbymchenry#1500) Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached us, never whether the match was evidence, and an absolute floor admits noise on any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses took 63% of this repo's own "how does explore allocate its output budget" answer on nothing but an unused `const explore` and a `const BUDGET`. Four levers: - KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5, variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge anywhere in the graph (`contains` excluded — nesting is not usage) drops to 0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's own edges answer most cases free. No measurable latency change (210 vs 211 ms/call, n=12 interleaved). - PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated symbols. - RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND the graph mass. Score alone would not have fixed colbymchenry#1500 — the generated CRUD carries MORE graph mass than the hand-written use-case, and graph mass outranks score in the comparator. Self-normalizing, never a hard exclusion. - RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength direct match so concentration elsewhere can never exclude one (without it a named-seed-heavy file pushed the floor to 21 and dropped a file the agent had named by class name). Backfills to 3 candidates when it would leave fewer, and drops the evidence requirement rather than return nothing at all. excludeLowValueFiles was dead config — declared per tier, read nowhere; the test/spec exclusion has been unconditional for a while. Removed. The real gap was the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir (express, cobra, most of npm and Go) never matched — express's routing question spent 59% of its envelope on three test files. Anchored at `^` too, and the filter now runs before the floor and judges "are there other candidates?" on the whole gather. Measured before/after on the same indexes (baseline bd86ad2): - payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go delivered 0 -> 38.9%. Generated ranks #3/#4, was #1/#2. - self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1. - express "route a request": 59% to test/* -> lib/application.js + lib/response.js - cobra x3, codegraph "indexing pipeline": byte-identical (control) Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this file score X" is legible. Selection stages reordered to match the pipeline. CG-6's gates flip from it.fails to live regressions except the byte-split ones, which stay open for CG-12 (allocation still follows file size within the ranked set). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ative cliff (CG-12, colbymchenry#1500) The explore envelope used to follow FILE SIZE, not relevance. Every admitted file was capped at the same flat `maxCharsPerFile`, while the whole-file rule handed anything under `maxCharsPerFile * 3` its entire contents — a 3x swing decided by how big a file happened to be: - self-query: `memory-budget.ts` (score 18) shipped whole and took 51.2% of the response; `src/mcp/tools.ts` (score 41, 4x the graph mass, 3x the term hits — it holds the allocator itself) was clipped at 3,800 and got 32.9%. - colbymchenry#1500 Go fixture: two generated CRUD files shipped whole at ~4.5K each AND consumed two of the tier's four file slots, so `BuildPayslip` — the hand-written "calculate" half of the question — ranked #6 and never rendered at all. `allocateExploreBudget` now reserves each ranked file a share of the envelope before anything renders, so the render loop spends a reservation instead of racing for whatever the files above it left: - weight = score x worth x (spine ? 2 : 1), where `worth` is `rankPenalty` applied a SECOND time — ranking answers "is this file about the query", allocation answers "will these bytes teach the agent anything", and generated CRUD can legitimately rank while its bytes stay boilerplate; - a relative cliff at 15% of the top weight (capped at SCORE_FLOOR_MAX, so a god-file can't silence peers the score floor just admitted) gives a file ZERO source — path, symbols and line numbers only — and crucially frees its `maxFiles` slot for a file that earns its bytes; - every admitted file gets MIN_CHARS, then the remainder splits by weight: the floor keeps a diffuse survey question returning a spread, the remainder concentrates a precise one; - the flat per-file cap is retired as the primary guard, leaving a 70%-of- envelope safety valve. Two changes were needed to make the reservation bite: an oversize cluster now shrinks by whole MEMBER symbol ranges (a single-cluster god-file previously took ~40% more than allotted, and the file below it was dropped for lack of room), and the arrival-order budget stops are gone — they cut files by the order they were reached rather than by merit. Measured: payroll-go answer group 25.6% -> 78.7%, generated 57.4% -> 0%, and `func (s *Service) BuildPayslip` now delivered; self-query `tools.ts` 18.5% -> 60.6%, past the epic's >50% bar. Controls hold: cobra/gin diffuse survey queries keep their file spread (3->3, 3->4), express's middleware query is byte-identical, and gin's flow query moves its top file from the thin `ginS` singleton wrapper to `routergroup.go`. One documented exception to "no previously-unclipped file becomes clipped": `memory-budget.ts` was unclipped-whole at 5,672 and now clusters within its 3.1K reservation. That is the epic's own diagnosis of the bug — it scored 18 against 58 and was taking the larger slice purely for being small. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…henry#1500) Coverage for the CG-12 allocator, built around "would this go red if the lever were removed" rather than line coverage — every way this regresses is silent, ending in an agent falling back to Read. Unit (`explore-proportional-allocation.test.ts`, 18 -> 38): calibration pins, envelope safety across every tier and 30 candidate shapes, the cliff boundary, spine weighting/trim survival, the diffuse control, and the degenerate inputs — identical scores, a lone file, a runaway top scorer, zero results, maxFiles 0, a non-finite score. End-to-end (`explore-allocation-e2e.test.ts`, new): CG-6's second regression fixture as a deterministic synthetic mirror — a large relevant file, a small helper that used to win by shipping whole, and an incidental `explore`/`BUDGET` collision — asserting per-file budget share, not file presence. Plus degenerate result sets and a survey-style diffuse control through the real render loop. The live self-query arm stays in probe-allocation.mjs, where drift is a number to re-baseline rather than a red suite. Reverting the render loop to the pre-CG-12 rules reproduces colbymchenry#1500 on the mirror exactly and takes 5 e2e + 2 payroll gates red: file score pre-CG-12 CG-12 src/mcp/allocator.ts 77.5 4,843 (39.7%) 9,335 (80.1%) src/util/budget-math.ts 36.0 6,079 (49.8%) 1,037 ( 8.9%) Two defects the invariants surfaced, both fixed in tools.ts: - rounded shares could sum past `pool`, so "reservations fit the envelope" was approximate rather than exact; both terms now floor - a non-finite score made every share Infinity/Infinity, handing the render loop a NaN allowance; `weightOf` now fails safe to 0 Also adds a hard-ceiling gate to the payroll fixture — at 19.3K against a 19.5K ceiling it is the only fixture that stresses the ~25K inline cap — and exports EXPLORE_ALLOCATION so invariant tests read the constants while one test pins the literals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…baseline A/B (CG-15) ab-new-vs-baseline.sh now builds and indexes once per arm and runs the task RUNS times (default 1), so the >=2-runs-per-arm rule costs one build instead of N. Both arms run with CODEGRAPH_NO_PROMPT_HOOK=1 — the machine's ambient front-load hook resolves to whatever is in dist/, a second uncontrolled channel that confounds the tool-call counts — and point explore's CG-4 diagnostic at a per-arm sidecar. parse-run.mjs gains --envelope/--answer: the per-file share of the explore source envelope, parsed from the rendered markdown so it works on ANY build. The CG-4 sidecar only exists post-CG-4, so it cannot measure the baseline arm; this is the only view that measures both arms the same way. Folded into parse-run.mjs rather than added as a new script on purpose: a new file named after explore's budget scores into the self-query fixture's own corpus and moved its answer share 59.9%% -> 47.9%%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Killing ab-new-vs-baseline.sh mid-baseline-arm left the engine checked out at the baseline ref with the post-baseline files deleted, so every later build in the working tree was silently the OLD code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…te fails on the control Three repos, both arms codegraph-on, sonnet/high, 3 runs per arm. PASS on the two medium repos: client-go (the reporter's Go shape, 2,001 of 2,454 files generated) and excalidraw hold Read 0 in every run of both arms, excalidraw goes 34s -> 24s at the median with one fewer explore call, and the generated clientsets/informers that took 10.5%% of a baseline envelope appear in no new run. FAIL on express, the small control, in 1 run of 3: 4 Reads and 52s against a baseline that read once. Not agent variance — replaying that run's query deterministically, lib/utils.js goes from 6,380 bytes whole to a 583-byte cluster stub and the envelope shrinks 13.8K -> 9.2K against an unchanged 13,000 budget. The diagnostic shows the allocator was right and the render loop was not: utils.js is the top-ranked file, was reserved 3,870 chars, and spent 583. The whole-file bound (allowance + grace = 4,450) lands just under the file's 5,293 bytes, so the whole-file render is declined and the unspent reservation is dropped rather than redistributed. Bar 1 is the hard gate, so per CG-15's acceptance rule the design goes back to CG-12 — the budget is not to be widened to compensate. Two candidate fixes are written up in the design doc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…acing shape (CG-16) Three separate engineer-shaped entries (CG-5 generated detection, CG-10 scoring, CG-12 allocation) become two user-facing bullets under Fixes, in the shape colbymchenry#1500 actually reported: explore concentrates on the code that answers the question, and a generated CRUD/protobuf layer no longer crowds out the hand-written code beside it. Per the CHANGELOG rules: strips the benchmark counts and percentages (the client-go 2,001-file count, the quarter-to-four-fifths envelope shift) and the internal symbol names, keeps the re-index note, and credits the reporter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lbymchenry#1500) A file whose proportional reservation lands below its own size stopped rendering whole, and the fallback cluster render could leave most of that reservation unspent — the bytes were neither delivered nor redistributed. Found by CG-15's agent A/B on the express control: `lib/utils.js`, the top-ranked file, was reserved 3,870 chars and spent 583. The whole-file grace bound (reservation + a sliver) sat just under the file's 5,293 bytes, so the whole render was declined and three matched symbols became a stub. The source envelope fell 13,849 -> 9,241 against an UNCHANGED budget, and the agent Read the file back four times in 1 run of 3. Two levers, per the task's candidate fixes: - WHOLE_FILE_BUY_FRACTION: a reservation that already covers 60% of a file buys the whole file. Funded from ONE shared overshoot pool sized at 15% of the envelope, spent in rank order. Per-file funding is the version that fails, and it fails the same way the bug does — the merit test is a ratio, so several files qualify at once and N independent overshoots push the last section past the render ceiling. Measured on the payroll fixture: three files bought whole and `payslip_builder.go` was dropped entirely. A dropped section is strictly worse than a clustered one. - Reservation carry-forward: what a file cannot spend goes to the next file down, bounded by MAX_SHARE. Tracked as two running totals rather than a `spent` variable threaded through the render loop's dozen exit paths, so no path can forget to account, and symmetric — a buy that overshoots suppresses slack until a later under-spend covers it. Express reproducer: `lib/utils.js` 583 -> 6,268 whole, envelope 9,241 -> 14,505 on the same 13,000 budget. The `memory-budget.ts` exception CG-14 documented is RESOLVED rather than re-justified: it ships whole again at 5,672 (27.3%) while `src/mcp/tools.ts` rises to 52.6% — so the answer file wins the envelope AND no previously-unclipped file is clipped, which is CG-12's own acceptance criterion finally holding. Two hermetic fixtures added, one per lever, because nothing in the suite had this shape — which is how it shipped. Both mutation-tested: removing the buy arm reddens 3, removing the carry-forward reddens 2, and removing the funding guard reddens 4 (including payroll's dropped `payslip_builder.go`). Their `fixture shape` blocks are load-bearing: the gates pass vacuously if a target ever drifts inside the grace bound, so the window is asserted directly. Full suite green (2,868 passed); both colbymchenry#1500 regression fixtures pass.
…cision) Records both levers, the funding-pool design (and the per-file version that dropped payslip_builder.go), the resolved memory-budget.ts exception, and the two hermetic fixtures with their mutation matrix. The CHANGELOG clause 'no longer trimmed while a smaller, weakly-related one is included whole' was imprecise after CG-21: the smaller file often IS still included whole now, when its share nearly covers it. Reworded to say what the fix actually guarantees.
…g (CG-21) Found reviewing the CG-21 fix rather than by a failing test, and it is the same defect inverted. A whole render that overruns `renderCeiling` is skipped ENTIRELY (the branch refuses to slice a file mid-method), so a buy that is approved by the funding pool but refused by the ceiling trades a clustered section for NO section. Only reachable on the 24K tiers. The funding line is `reservedTotal + 0.15 * envelope` — ~27.2K when a medium repo saturates — while `renderCeiling` is `min(1.5 * envelope, 25000) - 600` = 24.4K, so funding can approve ~2.8K the ceiling then refuses. At 13K the line is ~14.4K against an 18.9K ceiling and the two cannot cross, which is why the small-tier fixtures cannot see it. Failing the test in `buysWhole` drops through to the cluster path, which is bounded by `headroom` and always renders something. The GRACE arm is left alone deliberately: a file within a sliver of its reservation that still does not fit is genuinely at the end of a full response, and that predates this epic. Verified inert on the three A/B repos, so the agent A/B measured the same behaviour: excalidraw byte-identical on 3 queries, client-go byte-identical on 3 queries, express reproducer unchanged at 15,984. Full suite green (2,867 passed; one unrelated fs.watch timing flake that passes 30/30 in isolation).
Re-runs CG-15's agent A/B on the fixed build: same harness, same three prompts, same baseline ref, n=6 per arm on express and excalidraw. Read = 0 in all 15 new-arm runs. The express regression that routed the defect to CG-21 does not reproduce in 6 attempts, and the baseline now reads in 4 of 6 while the new arm reads in none (median 24.5s -> 21.5s), so the control beats the arm it previously lost to. client-go holds 92.7-96.2% answer share against a baseline run at 53.8%. Excalidraw's new arm is ~8s slower at the median and that is recorded as NOT attributable to the build rather than waved through: explore's own latency is 374ms vs 372ms on the same query and index, the deterministic responses differ by +2% with one byte-identical, and the unchanged main build's own median moved 34s -> 26.5s between the two sessions — the same magnitude as the gap. Bars were not re-baselined; they are CG-15's four, applied to a larger sample. The CG-15 section is kept intact and marked superseded, because its root-cause analysis is the record of why the fix looks like it does.
…henry#1500) CG-21 fixed the unspent-reservation defect and re-ran the A/B itself. CG-22 is the gate proper: CG-15's setup, unchanged, measured independently of the task that wrote the fix. RUNS=3, both arms codegraph-on, sonnet/high, CODEGRAPH_NO_PROMPT_HOOK=1 on both, baseline pinned to 49c11fc by SHA, fresh clones of the same three repos and the same three questions. All four bars pass. Read = 0 in all 12 new-arm runs (express 3, excalidraw 3, client-go 6) while the baseline reads in 3 of 3 express runs and 1 of 6 client-go runs; the express run that failed CG-15 with 4 Reads of lib/utils.js now reads nothing and receives the file whole. Answer share >= 66.6% in every new run. Medians: express 26s -> 24s, excalidraw 26s -> 26s, client-go 35s -> 36.5s at n=6 with fully overlapping ranges. Deterministic core re-measured on BOTH builds in one session rather than quoted: lib/utils.js renders whole at 6,380 B on baseline and on HEAD (583 B stub under CG-12), and the source envelope goes 13,849 -> 14,913 against an unchanged 13,000 budget, so the reservation is spent and the envelope stops shrinking. client-go's +1.5s median is attributed away from the build: explore latency 669 vs 668 ms (n=5) and the new build's deterministic response is both smaller (15.8K vs 18.9K) and more concentrated (top file 50.2% vs 35.7%). Two counter-points recorded as measured, not smoothed: excalidraw's new arm runs below its baseline on answer share (66.6-81.9 vs 75.5-92.7, all well over the bar), and this session's client-go baseline sampled well (85.6-100%), so the colbymchenry#1500 gap is smaller here than in CG-21's session. CHANGELOG: the two colbymchenry#1500 bullets were multi-sentence paragraphs carrying implementation detail. Rewritten to house style as four bullets that lead with the symptom, with the mechanism, the banner catalogue and the old-behaviour contrast dropped; the @LeDuyViet credit and the re-index note stay. Suite green on the measured build: 171 files, 2,868 passed, 6 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… sessions (CG-7) The A/B arms reported cost, tokens, time and tool counts for one headless question. They could not report what issue colbymchenry#1500 actually measured: how much of the context window a tool's responses still occupy once the question is answered, which every later turn is then charged for. parse-run.mjs now measures that. Tokens are measured, not estimated: for each assistant request, input + cache_read + cache_creation is the exact token count of its whole prompt, so consecutive requests differ by exactly what was appended between them. That delta is priced against the characters in the gap, calibrated on gaps that are >=80% tool result. Explore output lands near 2.3 chars/token, so the usual bytes/4 estimate would have under-counted it by ~40%. Content also leaves the window, so residual is tracked apart from contributed: a compact_boundary clears the resident set, and a mid-run context drop is micro-compaction, which sheds the oldest tool results first and is applied FIFO. run-all.sh takes "Q1||Q2||Q3" and runs them as one resumed session, one segment file per turn; parse-run.mjs stitches the segments back together. bench-readme.sh now runs each README repo as a three-turn session (CG_TURNS=1 restores the single-question form). parse-bench-readme.mjs reports the arms' retrieval residual side by side -- codegraph's responses against the without-arm's Read/Grep/Bash -- in absolute tokens, share of context, and share of window, and says so explicitly when the rows it aggregated were single-turn. Two transcript traps are handled and documented at the call site: Claude Code emits one assistant event per content block, all carrying the same usage (summing per event double-counts every turn with both thinking and a tool_use), and the streamed output_tokens is a partial snapshot. Occupancy lives in parse-run.mjs and is imported by the aggregator rather than extracted to a module -- a new scripts/agent-eval/*.mjs scores into the self-query fixture's own corpus and moves its numbers.
…residual (CG-7) The first request's prompt is system + tool schemas + the question, before any tool has answered, so differencing the arms' ctxBase prices what codegraph occupies whether or not the agent ever calls it. Measured on gin: +775 tokens, small because the tool is deferred -- only its name is in the initial listing.
On a gap that is >=95% one tool result, the measured context delta IS that result's token count, so the spread between it and the run-level ratio is the attribution error. Median over such gaps: +/-1-2% on real runs.
The suite passed unchanged with `CODEGRAPH_NO_REBIND=1`, so the larger half of CG-33 — the rebind pass — had no coverage at all. The cause was the ground truth, not the cases: `rebuildEdgeSet` called `indexAll()` on the live handle. That is not a rebuild. Every file hashes identical, so the store writes nothing (`nodesCreated: 0`), no reference is re-created, and every edge survives — the comparison read the synced index against itself and could never fail. It now goes through `CodeGraph.recreate`, which deletes the database file the way the CLI's `index` command does. With a real rebuild, three existing cases fail under the kill switch. Adds two more for the rules that carry the risk: - an edge with no `refName` stamp (older engine) and a synthesized (`provenance='heuristic'`) edge are never deleted — both planted directly, and each verified load-bearing by mutation; - a name over the 500-edge ceiling is declined losslessly rather than rebound in part, with a rare name in the same sync as the control that proves the pass ran. The per-file-vs-batch-wide delta rule is likewise confirmed by mutation: a batch-wide name set fails its case. CODEGRAPH_NO_REBIND=1 now fails 4 cases; unset is green; full suite green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nners (CG-25) Cloudflare Wrangler's `worker-configuration.d.ts` (~12k lines of ambient types) carried no banner any GENERATED_CONTENT_PATTERNS entry matched: every existing marker requires `DO NOT EDIT`, a standalone `@generated`, `<auto-generated>`, or the literal `automatically/auto-generated by` phrasings. Wrangler emits a bare `Generated by Wrangler by running `wrangler types``, so the file ranked with pen 1.00 and won 79.4% of an explore envelope on generic token overlap alone (CG-24). The discriminator is the reproduction instruction, not the word "generated": the banner must name a tool AND then say `by running`, i.e. two separate "by" clauses. That keeps prose out — "the nightly summary is generated by running the ETL job" has only one — while catching every CLI-driven emitter that tells you how to regenerate. Precision swept over 441,856 files across the whole local source tree: 5 hits, all genuine Wrangler output, no false positives. Isolated before/after on the CG-24 repro (same query, same index, only the `files.generated` flag differing): before pen 1.00 score 115.0 share 79.4% 3 files rendered after pen 0.30 score 35.4 share 21.1% 4 files rendered The new pattern stays in the existing table position, below the header window the detector scans, so the module still does not classify itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…act budget (CG-30, CG-31, CG-26) Lands the three-branch allocation stack. Every admitted file now receives at least its reservation before any file draws on carry-forward slack, on every render path — cluster, whole-file GRACE, and whole-file BUY. CG-30 bounded how far an oversize cluster member may overshoot (windowed on whole lines past 1.5x rather than emitted whole or dropped). CG-31 gave the cluster path the `owedBelow` displacement guard the BUY arm always had, holding back only the prefix of what is owed below that the response can actually pay. CG-26 closed the three remaining holes: the whole-file arms had no displacement guard at all, section overhead was charged at a flat 200 against a real 300-500, and `owedPayableBelow` held all-or-nothing where it should hold partially. Deterministic across the 6-repo suite, clean-rebuilt indexes, both builds: no repo truncates, no repo loses a file, okhttp gains one, and every repo lands at or under the 25,000 hard ceiling. Accepted trade (maintainer decision): excalidraw -552 and okhttp -164 source chars against the CG-31 tip, in exchange for the trailing pointer list surviving instead of being discarded whole. Those bytes existed at the CG-31 tip only because it over-filled a ceiling it mis-measured and then dropped the entire epilogue; a pointer the agent can act on beats a few hundred chars on the last-ranked file. Two issues opened during this work were closed as invalid rather than fixed: CG-32 (named-file ordering) and CG-34 (allocator over-reservation). Both were filed on diagnoses that did not survive measurement — CG-32's symptom was index drift (CG-33), and CG-34's premise was overturned by CG-31's own results. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A generated Cloudflare Wrangler ambient-types file was not flagged generated, so it ranked with no penalty and competed with hand-written source on generic token overlap. The banner shape it uses — "Generated by <tool> by running <command>" — matched none of the existing content patterns, all of which require DO NOT EDIT, a standalone @generated, or the "auto(matically) generated by" phrasings. Precision is held by requiring TWO 'by' clauses: the banner must name a tool and then say 'by running'. Ordinary prose ("the report is generated by running the nightly job") has only one and does not match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e (CG-27)
A file whose top-level symbol spans almost all of it — createFoo() returning
an object of closures — is how Svelte 5 rune stores, React custom-hook modules,
IIFE module-pattern JS and Zustand's create((set,get)=>({…})) are all written.
probe-factory-closure.mjs measures what such a file DELIVERS from within: which
inner symbols' definitions reach the agent, not how many bytes did.
…(CG-27) CG-27 asked whether the >50%-of-file envelope drop should cover `function` / `method`, so a `createFoo()` factory returning an object of closures stops merging every closure inside it into one cluster. Measured on a hermetic fixture, it should not, and the issue is closed as obsolete with CG-30 credited. Two mechanisms already absorb the shape. shrinkCluster orders members by (importance desc, size ASC) and refuses any member that overruns the cap once something is kept, so a file-spanning member is only selected when it is the sole member of the top importance tier — eight of nine query shapes never selected it at all. When it IS selected, CG-30 windows it on whole lines, so the file still delivers bounded, readable source (6 of 9 closure definitions in that configuration). Dropping the range instead SPLITS the file, and only the first-chosen cluster may be shrunk: a trivial 7-line cluster won the density tiebreak and the answer-bearing cluster was dropped whole — rank-#1 file 7,539 chars and 7 of 11 closures to 397 and none. Reaching the same intent more carefully (defer the envelope MEMBER inside shrinkCluster, leaving clustering untouched) is noise: 69 vs 68 closure definitions across nine query shapes. Nothing shipped. Adds the fixture, the probe, a standing gate on the outcome, and the record — including a real defect the measurement exposed on the epic tip: django's query.py leaves 8,212 of 10,135 unspent and drops a score-290 cluster to keep a score-14 one. Filed separately. No behaviour change, so no CHANGELOG entry.
CG-27 proposed adding function/method to ENVELOPE_KINDS so a factory closure spanning most of its file stops merging every inner symbol into one cluster. The issue required the ranking claim be measured before any fix. It was, on a hermetic fixture built to make the pattern maximally visible, and it does not hold — nothing shipped to src/. The literal change is a large regression: dropping the enclosing range SPLITS the file into a trivial cluster (a type alias plus a helper, span 7) and the answer-bearing one (every closure, span 359). Cluster ranking breaks the equal maxImportance tie on density, so the trivial cluster wins, is taken first, and is the only one that may be shrunk; the answer-bearing cluster then does not fit and is dropped whole. Rank #1 fell from 7,539 delivered chars to 397, and from 7 of 11 inner closures to 0. The enclosing range was holding the file together as one cluster, inside which shrinkCluster already did the per-symbol ranking the issue asked for. A better mechanism reaching the same intent — deferring the envelope member inside shrinkCluster, leaving clustering untouched — is noise: 69 vs 68 inner definitions across nine query shapes, one better, one worse, seven unchanged. The one configuration where the envelope IS selected (the factory as sole top-tier member) is already absorbed by CG-30, which windows it on whole lines: a contiguous readable head carrying 6 of 9 closures, bounded and never empty. Kept: the fixture, the deterministic probe, and the measurement record. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A file that declares nothing but types and that nothing in the index depends on — a hand-written ambient `.d.ts` of global shims, vendored typings, module augmentation — cannot answer a flow question: no bodies, no call edges, no behaviour, nothing typed by it. But the identifiers it declares are exactly the generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the implementation. Measured on the new fixture: rank #1 and 51% of delivered source, with the flow's own entry file pushed out of the response entirely. Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that opened this is already handled by CG-25's banner detection, worth 15-46 points of envelope share across four flow queries. CG-25 credited; only the un-bannered case needed anything. `rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken as the STRONGER of it and the generated penalty rather than multiplied — one property two signals see must not be charged twice. Detection is structural, not by extension, and four conditions deep. Two of them were forced by measurement: requiring every symbol to be type-level takes the corpus flag rate from 1-18% (which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's locale tables) down to 0-4%; requiring that nothing depends on the file separates an ambient shim from a working types module, and without it the rule demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate. A query that NAMES a declared type is exempt, so a question about a type still reaches its declaration at full weight. Precise tokens only, so "…the file body…" cannot exempt a `Body` interface it never meant to name; this needs its own set because `namedSeedIds` is callable-only and a type never becomes one. Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md: 6-repo envelope sweep byte-identical against a clean baseline build, zero ambient files reach the candidate set on VS Code across five queries, corpus flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…CG-28) Both halves of the issue were measured on a hermetic fixture of four declaration-shaped files varying on banner and depended-on-ness. CG-25 already handles the motivating file: the Wrangler worker-configuration.d.ts that opened this issue is demoted by the generated penalty alone, worth 15-46 points of envelope share across four flow queries. No new mechanism for it. The narrower gap is real. A declaration file with NO banner carried pen 1.00, took rank #1 and 51% of delivered source on a prose flow query, and displaced the flow's own entry file out of the response entirely. The rule is deliberately narrow, and both conditions were derived by survey rather than guessed. 'Declares no callable and calls nothing' flags 1.1-18.0% of files across the corpus and catches real source — okhttp's SocketPolicy.kt, tokio/src/runtime/mod.rs, Alamofire's umbrella file, django's locale format tables. Requiring every symbol to be type-level drops that to 0-4%. The 'nothing depends on it' condition was added after the broader version demoted a pure-interface file with 13 inbound imports and broke the CG-31 displacement gate — a different invariant entirely. Does NOT stack with the generated penalty: rankPenalty takes Math.min of the two, so a file that is both takes the stronger, never the product. A query that NAMES a declaration symbol exempts its file entirely, so asking about a type still reaches it at full weight. Six-repo envelope is byte-identical to the pre-change tip — the rule does not fire on any benchmark repo, consistent with the 0-4% survey. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four shipped fixes, one open defect (CG-36), and five issues closed because measurement contradicted them. The headline is that the reported symptom was not an explore bug at all — it was a degraded index (CG-33), and the reported query answers correctly on a clean rebuild with no explore change. Records the two traps that cost real time and are now guarded in tooling: the nonexistent .codegraph/graph.db path that sqlite3 silently creates, and ab-new-vs-baseline.sh swapping src/ mid-run so a commit captures baseline sources. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…opping it (CG-36) A file's ranked clusters were all-or-nothing past the first one: the top-ranked cluster was taken (shrunk to fit when it had to be) and every cluster below it was rendered whole, then either fit the remainder or was dropped entirely. On a file whose top-ranked cluster is TRIVIAL that discards the answer — django's `db/models/sql/query.py` kept a 22-line glue cluster and dropped the 624-line `Query` body, spending 1,923 of a 7,947 reservation; okhttp's `RealInterceptorChain.kt` did the same behind its import header. The response stayed full, which is why this was invisible: the unspent reservation carried forward exactly as designed and a file scoring a fifth as much took the bytes. Two sites, the same rule — hold the remainder while it is still worth a section (CG-26's between-FILES lesson, applied between CLUSTERS): - selection now shrinks a later cluster into what is left of the file's budget, by the same whole-member rule the first cluster already used; - the ceiling trim re-renders the weakest cluster into the room that remains before dropping it. On excalidraw's `typeChecks.ts` the section-cost estimate missed by 13 chars and a 1,512-char cluster — the file's highest-SCORING one — was thrown away to pay for it. Cluster RANKING is untouched: measured, both real cases lost on `maxImportance`, not on the density tiebreak the issue suspected, and density-first is what keeps Alamofire's `Session.swift` from burying its methods under the property list. Suite (6 repos, clean-rebuilt indexes): all 8 starvation flags cleared, +1,012 source chars net. django's `sql/query.py` 1,923 -> 10,082 of 7,947, okhttp's `RealInterceptorChain.kt` 1,474 -> 6,038 of 6,058, gin's `routergroup.go` 3,273 -> 5,632. okhttp trades its rank-6 file (score 21) for +7,196 chars in the two files that answer the question. Ships two fixtures pulling in opposite directions (`starved-cluster-ts` and `dense-header-ts`), a `spendShareAtLeast` gate in probe-allocation, and probe-file-spend.mjs — a standing per-file reservation-vs-delivered sweep.
The issue blamed the density tiebreak; both real cases lost on maxImportance, so ranking was left alone. Full before/after table, the one cost (okhttp's rank-6 file, squeezed out by reservations that were already structurally over-subscribed), and what ships to keep it measurable.
…it (CG-36) A file whose top-ranked cluster was trivial kept it, dropped the cluster carrying the answer WHOLE, and left most of its reservation unspent — because only the first-chosen cluster could be shrunk. CG-31's carry-forward then correctly handed that slack down the rank order, so the budget was not merely unspent but REDIRECTED to weaker files. django's sql/query.py (score 83, reserved 7,947) went from 1,923 delivered chars to 10,082, and its envelope share from 7.7% to 40.4%; contrib/admin/ filters.py (score 18) went from 8,057 at 355% of its reservation down to 2,198 at 97%. All 8 starvation flags across the suite clear. Net +1,012 source chars. The issue named the wrong fix point and the measurement said so: both real cases lost on maxImportance, NOT on the density tiebreak the issue and its duplicate (CG-37) suspected. Cluster ranking was left untouched, so the Session.swift case density-first exists for still works — now pinned by a dense-header fixture. Accepted cost: okhttp trades its rank-6 file (score 21, reserved 1,999) for +7,196 chars in the two files that answer the question, taking it from 6 delivered files to 5. django -159, okhttp -219 and tokio -25 source chars against the epic tip; gin +1,176, alamofire +187, excalidraw +52. django also stops cutting its epilogue. Ships probe-file-spend.mjs, a standing suite-wide probe for reservation vs spend, so this stays measurable — the original evidence came from ad-hoc instrumentation that no longer existed and had to be re-derived by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The epic record said nothing was open. CG-38 is: agent-named symbols in the tail of a large file never render, which the epic's probes cannot see because none of them measures whether the named symbol appeared. Also corrects a wrong claim made while investigating it. The epic was said to have regressed its own motivating query; that comparison varied the index as well as the engine. A controlled bisect holding the index fixed shows the pre-epic engine rendering 12 lines and CG-36 rendering 463 — the epic strictly improves the case, and the symbols render at neither. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CG-24: explore response noise — allocation fixes, generated-file detection, and index-drift convergence
…ts (CG-38) `codegraph_explore` never returned `queueMessage` (L1087) or `flushQueuedMessages` (L1102) from a 1,414-line file, on a symbol bag or a prose question, even with that file at rank #1 holding 67% of the envelope — the agent got a same-stem `QueuedMessage` interface at L70 and had to Read the file for the functions it had named. Pre-existing at every build including pre-epic (controlled bisect, index held fixed). Two independent causes: 1. `buildFlowFromNamedSymbols` returns the Flow prose AND the set of node ids the agent named — and the latter is the whole guarantee, since it injects a named def into its file's cluster ranges at importance 9. Its bail-outs returned EMPTY, zeroing the identity whenever there was nothing to PRINT. Two sibling closures that never call each other produce no chain, no synth hop and no boundary, so both defs lost importance 9 and the file rendered from its head. `identityOnly()` now separates the two, gated on shape-precise tokens so a prose word that exact-matches a callable cannot promote itself. 2. The ceiling trim filled in SOURCE order, so an over-ceiling render always dropped the END of a large file first. The shrink HAD kept both symbols (1022-1121); the trim cut back to 839. `windowToCeiling` now takes the spine call site plus every importance>=9 member as focus lines, tries the full ceiling first, and splits the held-back reserve evenly with carry-forward — greedy-in-source-order reproduced the bug one level down. The shrink's loose size estimate is left alone deliberately, and the comment now says why: making it exact was built and measured WORSE (it stops at the last member that fits whole and the released bytes carry forward to lower-ranked files, costing payroll-go's `s.store.Upsert`). `bound()` clamps to the ceiling anyway, so the slack costs no bytes; it just must not pick the survivors, which is what the trim now handles. The measurement gap this closes: every existing probe is aggregate — envelope share, per-file spend, source totals, file counts — and all are green on a response that returns 25K from the right file and omits the named function. `probe-named-symbol.mjs` checks the definition LINE against the response's rendered lines, per symbol. Suite envelope byte-identical to main on all six repos; probe-allocation 4/4, no starvation flags; 180 files / 2,997 tests green. Fixture: 7/7 fail on main, 7/7 pass here, deterministic over 4 runs per arm.
CG-33/CG-35: converge incremental sync with a full rebuild
CG-38: guarantee an agent-named symbol renders, wherever it sits
fix(telemetry-dashboard): accept Origin: null on login — no-referrer policy locked Chromium out (CG-16)
…lbymchenry#671) Recut of colbymchenry#678 against the current instructions — the original predated the explore-first rewrite and conflicted in both files it touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…supported-languages feat(mcp): surface supported languages in MCP server instructions (recut of colbymchenry#678)
# Conflicts: # CHANGELOG.md
…nstaller-targets feat(installer): GitHub Copilot targets — VS Code, Copilot CLI, JetBrains
…nry#1515) Making unions first-class nodes leaves the third loss in colbymchenry#1515 open: interfaceOverrideEdges enumerates its concrete side as ['class','struct'], so a union implementor is skipped even though it now has a real node and a real `implements` edge. "Who implements this trait" then answers wrongly rather than incompletely — the struct beside it bridges and the union does not. Add 'union' to that tuple, plus a regression test that pins the Rust trait -> union-impl hop (the struct implementor is the control proving the synthesizer ran). Verified the test fails on the union assertion alone before this change. No EXTRACTION_VERSION bump: main is already at 25 against v1.5.0's 24, so existing indexes are flagged stale for the next release regardless, and over-bumping is what turns the re-index hint into noise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolves the CHANGELOG conflict — main and this branch each prepended a bullet to [Unreleased] > Fixes; both are kept. Everything else auto-merged, including src/mcp/tools.ts, which main reworked heavily for the explore allocation/displacement work (CG-28/31/36/38) while this branch added the `union` kind to its container sets. Verified on the merged tree with the native kernel built: 3070 passed, 9 skipped, 0 failed.
…tions-not-indexed fix(c,cpp,objc,rust): index union declarations as a first-class `union` node kind (colbymchenry#1515)
Upstream has not tagged a release since v1.5.0, but main carries 104 commits we want ahead of that tag - most relevant to FM-Agent: - incremental sync no longer drifts from a full rebuild (replaying 80 commits through `sync` left 5.7% of connections wrong upstream, now 1.3%) - codegraph_node / codegraph_explore no longer slice current file bytes at outdated line positions after a file changed on disk (colbymchenry#1474) - C/C++/Objective-C/Rust unions are indexed as first-class nodes (colbymchenry#1515) - Python classes used as values produce reference edges (colbymchenry#1478) - the write-ahead log no longer grows without bound after a force-killed process, and the watchdog no longer kills a healthy server on slow disks (colbymchenry#1431) Pinned at c6aaa20 rather than tracking main, so the base stays a fixed point we can name. Merge only - no fork-layer changes. The fork layer applied cleanly with no conflicts: install.sh, install.ps1 and src/upgrade/index.ts still point at fmagent-project/codegraph, the release workflow is still the GitHub-Releases-only trim, and no upstream commit touched them. The version bump and the FORK.md base note follow in the next pull request.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 38d1f9733a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| readonly docsUrl = 'https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-mcp-servers'; | ||
|
|
||
| supportsLocation(loc: Location): boolean { | ||
| return loc === 'global'; |
There was a problem hiding this comment.
When users choose --location=local, this target is skipped even though Copilot CLI now supports project-level MCP files: GitHub's docs say it "looks for project-level configuration" in .mcp.json and .github/mcp.json (docs). As a result, codegraph install --target=copilot-cli --location=local leaves repo-local Copilot CLI sessions unconfigured; this target should write the local MCP entry instead of rejecting local installs.
Useful? React with 👍 / 👎.
| .nodes_meta | ||
| .iter() | ||
| .position(|m| m.name == type_name && matches!(m.kind, "struct" | "enum" | "class")) | ||
| .position(|m| m.name == type_name && matches!(m.kind, "struct" | "union" | "enum" | "class")) |
There was a problem hiding this comment.
This adds Rust impl Trait for MyUnion extraction, but the resolver's SUPERTYPE_BEARING_KINDS set still excludes union, so getSupertypes() filters these union nodes out before following their new implements edges. In Rust code where a union relies on a trait/default method through the conformance pass, the edge is stored but chained or inherited-method resolution still behaves as if the union has no supertypes; add union to the resolver's supertype-bearing set with this extraction change.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream has not tagged a release since v1.5.0, but
maincarries 104 commitspast that tag. Pinned at
c6aaa20358cd6adcd04b87bdef8e5803ad146f3arather thantracking
main, so the base stays a fixed point we can name.Why now
Five of those commits matter to how FM-Agent uses the graph:
syncno longer drifts from a full rebuild — replaying 80 commitsthrough
syncleft 5.7% of connections wrong upstream, now 1.3%. Call edges arewhat FM-Agent's layering follows.
codegraph_node/codegraph_exploreno longer slice current file bytes atoutdated line positions after a file changed on disk (Stale index + fresh disk read: codegraph_node/codegraph_explore return a DIFFERENT symbol's code under the requested name, while asserting "verbatim, current on-disk source … do not Read" — verified through a real serve --mcp colbymchenry/codegraph#1474)
uniondeclaration previously produced no symbol at all (Union declarations are not indexed in C, C++, Objective-C, or Rust colbymchenry/codegraph#1515)
return SomeSerializer,handler = SomeClass,registry dicts) now produce reference edges (Python: bare class references (return SomeClass, x = SomeClass, registry dicts) produce no references edges to classes colbymchenry/codegraph#1478)
and the watchdog no longer kills a healthy server on slow disks (Main thread unresponsive for ~60s ΓÇö killing the wedged process so a fresh one can start (#850). Disable with CODEGRAPH_NO_WATCHDOG=1. colbymchenry/codegraph#1431) — the
daemon is SIGTERM'd several times per FM-Agent run
Fork layer
Merge only, no fork-layer changes. It applied with no conflicts — no upstream
commit touched any of it. After the merge the fork differs from upstream in
exactly eight files:
Still no patch under
src/— the base remains byte-identical to upstream.Verification
npm run buildpasses.npm test: 2901 passed, 0 failed (163 files, 15 skipped).Merge with a merge commit, not a squash, so upstream stays an ancestor.