groundedness: gaps and pre-existing defects surfaced by the #147 review

[jerelvelarde]

Collected from the review of #222 so they don't get lost. None are introduced by that PR; it fixed its own blocker plus the two test-quality findings. Grouped by whether acting on them widens the gate (risks withholding correct answers) or narrows it.

Reminder of the stakes: after #143 the identifier signal is the only mechanism that withholds a response. A false positive here withholds a correct answer from a real reporter.

A. Spellings that escape the gate (widening — needs a false-positive measurement pass)

All verified returning []:

• Undotted kebab — `copilot-ghost-panel`. The dotted form is covered; the bare form isn't.
• camelCase invented names — `copilotFabricate()`, `copilotInvented`.
• Lowercase after the prefix — `useCopilotfabricated()`, `<Copilotinvented />`. Plausible model typos.
• Casing asymmetry worth at least a doc line: Copilot_Chat is an identifier because _ is in the character class, but copilot_chat isn't; COPILOT_API_KEY isn't while COPILOTKIT_TOKEN is.

Each of these widens the gate, so each carries false-positive risk. They want one deliberate pass with a corpus, not four separate regex tweaks. The strings above are the starting test matrix.

B. The retrieval-quality trade (decide, don't drift)

The widened gate can't distinguish a real API name from an invented one, so its false-positive rate is bounded entirely by retrieval quality. Measured: Register it with `useCopilotAction()` and mount `<CopilotSidebar />`. against a retrieved CopilotChat page not literally containing those strings → suppress = true.

So the five-ish names a correct answer cites most often are the ones most likely to be withheld. An allowlist of those five was considered and rejected in #222 — it doesn't help the #147 case (the fabrication isn't on the list either), and it converts "withheld a correct answer" into "published an invented one".

The actual fix is retrieval: if CopilotChat's page doesn't contain useCopilotAction, an answer citing both is standing on a source that doesn't support it. Tracking that here rather than papering over it in the scorer.

C. Pre-existing defects (narrowing or correctness — safe to fix)

• IDENTIFIER_PATH accepts a leading # but the split strips only . — `#copilotKitPanel` may be recorded with the # attached and dedup separately from its dotted form, turning one invented name into two entries and clearing the suppression bar alone. Same root cause means ID selectors can never ground.
• BARE_HOST_PATTERN's TLD list includes app|dev|io|co|sh — so `copilotKitBridge.io` is erased as a hostname before extraction sees it.
• The "never throws" docblock is wrong — a null sources, or a null element in the array, throws a TypeError inside the publish/withhold decision.
• The hedge reason quotes total hedgeCount while only excessHedges was charged.
• Backticked content over 80 chars is silently skipped — undocumented and untested.
• s.sourceUrl is in the grounding haystack, so any identifier that is a substring of docs.copilotkit.ai grounds automatically. Believed deliberate; needs a comment either way so the next reader doesn't re-flag it.

D. Test-quality (two more found, both may pass vacuously)

• The "claim wording never withholds" assertion appears wrapped in an if — which would make it pass vacuously the moment extraction starts inventing identifiers, i.e. exactly the regression it exists to catch.
• The name-claims test passes CHAT_DOCS and asserts neither coverage nor empty sources.

Two others from the same review — the non-discriminating "English word" corpus row and the unpinned ^ anchors — were fixed in #222.

[NathanTarbert]

Part of this landed; recording what remains so the issue is not read as wholly open.

Fixed: ID-selector names no longer keep their leading #, so a name written #copilotKitPanel can ground against a source that writes it bare (packages/outpost/ai/src/groundedness.ts, token.split(/[.#]/), with the rationale in the comment above it). The sourceUrl comment was corrected in the same pass.

Still open — section A in full. COPILOTKIT_IDENTIFIER_SHAPES is unchanged, so identifiers that do not match its shapes are never checked at all: a bare kebab-case component name, a lowercase-prefixed call, and a mis-cased hook or component all fall through. BARE_HOST_PATTERN still carries a broad TLD alternation. The "never throws" docblock still precedes an unguarded sources.map. The hedge reason still quotes the total hedge count while only the excess was charged.

None of that is affected by the # fix, so the scope here is section A rather than the whole review.