context-budget: two open measurements — interactive-session deferral eligibility; deferred-tool count_tokens billing

[kyle-sexton]

Follow-ups graduated from the context-budget topic slice (PR #2932) via the work-item tracker seam. Both are measurements the plugin's catalogue currently words around; each has a designed probe that could not run in the authoring environment.

1. Interactive-session deferral eligibility

Headless measurements stamp sessionKind: headless because deferral eligibility is partly server-decided and interactive sessions may compose the payload differently. The authoring container had no TTY, so this was never differenced.

Probe design: on a machine with an interactive terminal, run /context in an interactive session and diff its category table and deferred bucket against a headless measure.mjs snapshot taken from the same cwd, same binary, same settings, back to back. A divergence in which tools sit deferred (or in bucket totals beyond Messages jitter) confirms session-kind sensitivity; identity lets the report drop the caveat for that version.

2. Does the API bill for deferred-but-never-loaded tool definitions?

Decides how the report words the deferred bucket's cost ("recurring request weight" is currently justified by the documented send-semantics of defer_loading, not by a billing measurement). Blocked in the authoring container: no API credential (x-api-key header is required from the gateway).

Probe design (two count_tokens calls): identical requests except one carries an extra tool definition marked defer_loading: true (tool-search beta header on both). If the count rises by the tool's schema weight, deferred-but-never-loaded definitions are billed input; if not, the deferred bucket costs context-window-eligibility only and the report wording should soften.

Context

• Measured background: PR feat(context-budget): ship the startup-payload measurement plugin (0.6.0) #2932's pre-prune evidence record (docs/topics/context-budget/FINDINGS.md at the pre-prune SHA named in that PR's body); durable method and citations live in plugins/context-budget/skills/audit/reference/.
• Related resolved-by-measurement items (same session, v2.1.232 headless): PreToolUse ask fires and blocks under bypassPermissions; Streamable HTTP MCP tools defer into a dedicated MCP tools (deferred) category; denying ToolSearch is an anti-lever.

[kyle-sexton]

[kyle-sexton]

This was generated by AI during triage.

Triage notes

Verified. Checked the current wording this item describes against the live source (not the pre-prune record, which no longer exists in this checkout): plugins/context-budget/skills/audit/reference/report.md still stamps every record sessionKind: headless and still frames the deferred pool as "recurring request weight outside the window" — the exact caveats this item says are worded-around rather than measured. No commit since PR #2932 touches plugins/context-budget/, so neither open measurement has been resolved. Redundancy check: no existing probe, script, or recorded result for either question found under plugins/context-budget/. No related open intake to cluster with.

Agent Brief

Type: Task
Summary: Run two designed-but-unexecuted measurement probes for the context-budget plugin's audit report, then update the report contract's wording (and the lever catalogue / engine reference if the measured behavior affects them) to state the confirmed fact instead of the current worded-around caveat.

Why human-gated: both probes have a hard capability blocker for an unattended agent, not an open design decision — the routing test sends capability-blocked work (no interactive terminal, no external API credential) to human-gated rather than decision-defaulted.

• Probe 1 needs a real interactive TTY session (/context run interactively, not headless) diffed against a headless snapshot from the same cwd/binary/settings. An unattended/background agent runs headless by construction, which is the same reason the authoring session couldn't do it.
• Probe 2 needs a direct Anthropic API credential (x-api-key) to place two count_tokens calls. The authoring container had none; an autonomous work-loop agent should not be assumed to hold one either.

Current behavior:
The audit report's Stamp section always states sessionKind: headless and the Headline section always frames the deferred tool pool as "recurring request weight outside the window" — both are honest hedges standing in for two unmeasured facts:

1. Whether an interactive session's /context category table and deferred bucket diverge from a headless measure.mjs snapshot taken from the same cwd/binary/settings.
2. Whether the Anthropic API bills input tokens for a tool definition marked defer_loading: true that is never actually loaded into context, versus that definition costing only context-window eligibility with no billing impact.

Desired behavior:
Both probes run to completion and the report/engine/lever wording is updated to reflect what was actually confirmed:

1. Interactive-session probe. On a machine with a real interactive terminal: run /context in an interactive session and independently run measure.mjs snapshot (headless) from the same cwd, same Claude Code binary, same settings, back to back. Compare the category table and the deferred-tool bucket membership between the two. If deferred-bucket membership differs, or bucket totals diverge beyond ordinary Messages-category jitter, session-kind sensitivity is confirmed and the report's sessionKind stamp and any session-kind-dependent wording should say so explicitly (not just record the stamp value). If no divergence, the report can drop the implicit "this might differ interactively" hedge for the measured version.
2. Deferred-tool billing probe. Using a real Anthropic API credential, issue two count_tokens requests that are identical except one includes one extra tool definition with defer_loading: true (both requests carry the tool-search beta header). If the token count rises by that tool's schema weight, deferred-but-never-loaded definitions are billed as input — the report's "recurring request weight" framing is confirmed and can drop its "justified by documented send-semantics, not measurement" caveat. If the count does not rise, the deferred bucket costs only context-window eligibility, not spend, and the report wording should soften accordingly (e.g. from "recurring request weight" to something that doesn't imply per-request billing).

Key interfaces:

• The measurement engine's headless snapshot mode — the headless half of probe 1; already emits the sessionKind stamp plus the category table and deferred-bucket membership the interactive run gets diffed against.
• The audit report contract's Stamp and Headline sections — the two whose wording is contingent on these two results.
• The lever catalogue and its engine reference — check whether either result changes any lever's honesty category, detection method, or citation; update only if the measured fact requires it.
• The count_tokens API endpoint with the tool-search beta header and a defer_loading: true tool definition — the mechanism probe 2 measures directly, not through the plugin's own code.

Acceptance criteria:

• Probe 1 executed: an interactive-session /context category table and deferred-bucket membership are diffed against a same-cwd/binary/settings headless snapshot, and the divergence result (present or absent, with what changed if present) is recorded.
• Probe 2 executed: two count_tokens calls differing only by one defer_loading: true tool definition are compared, and whether the count rose by that tool's schema weight is recorded.
• The report contract's Stamp and Headline section wording is updated to state the confirmed fact for whichever probes completed, replacing the current worded-around hedge language with a version-scoped conclusion (do not claim permanence — measurements are point-in-time per the plugin's own recheck-trigger convention).
• If either probe could not be run (no interactive terminal available, no API credential available), the item is left open with the specific blocker restated rather than closed or silently dropped.

Out of scope:

• Re-deriving or re-verifying any other lever/finding already recorded in the lever catalogue — only the two named unknowns.
• Changing the measurement engine's methodology for headless snapshots; probe 1 only diffs interactive against the existing headless method, it doesn't redesign either.
• Any change to billing/pricing behavior itself — this only measures and documents current behavior.

[kyle-sexton]

Takeover: the prior work-item-lease v1 marker (holder kyle-sexton, session MELO-LAP-001-triage-202608230449, renewed 2026-09-05T01:21Z, ttl_hours=1) expired ~40 hours ago with no renewal. Taking over for a gate-audit pass.

This was generated by AI during triage.

---

Generated by Claude Code

[kyle-sexton]

This was generated by AI during triage.

Decision-ready summary — gate upheld: your call plus a capability gate this lane cannot clear

Lane: autonomous triage, instance vm-triage-202609050304. No labels changed. This comment exists so the gate can be cleared in one pass rather than re-investigated.

Recommendation

RECOMMENDED — keep needs-human. You are assigned to this one directly, and independently of that, neither probe can run in this lane.

Two reasons, and they stack:

1. Ownership. This item is assigned to you. The measurement design and how the report's wording changes on each outcome are yours to rule on.

2. Capability gate. Both probes are blocked on inputs no cloud triage lane has:

   • Probe 1 (interactive-session deferral eligibility) requires an interactive terminal to run /context and diff it against a headless measure.mjs snapshot taken back to back from the same cwd, binary and settings. This lane has no TTY, which is the same constraint that blocked the authoring container and is the reason the caveat exists at all.
   • Probe 2 (deferred-tool count_tokens billing) requires a live API credential. The authoring container failed with x-api-key header is required from the gateway; this lane is in the same position.

The probe designs themselves are well-specified and need no further work. What they need is an environment, not a decision about method.

What clearing the gate looks like

Route to a lane that runs on your local machine with an interactive session and an API credential, and both probes execute as written. On each outcome the report edit is mechanical and delegable:

• Probe 1 identity → drop the sessionKind: headless caveat for that version. Divergence → record which tools moved between deferred and loaded.
• Probe 2 count rises by the tool's schema weight → deferred definitions are billed input, and "recurring request weight" stands. No rise → soften the deferred bucket's cost wording to context-window eligibility only.

Neither edit needs a human once the numbers exist. The gate is on producing the numbers.

---

Generated by Claude Code

[kyle-sexton]

work-item-lease reclaimed: lease expired (renewed_at 2026-09-27T06:24:11Z) with no activity.