feat(slack): track the Slack bot's AI token usage for billing

[JeremyFunk]

The Slack bot runs its own model (Cloudflare Workers AI, on Railway) outside every path the Maple API already meters, so Slack-driven AI spend was invisible to billing entirely — no Autumn events, no persisted token counts, nothing.

Architecture

The bot reports usage; the API bills it. Same split as chat-flue → AiTriageWorkflow.

eve `step.completed`  →  agent/hooks/token-usage.ts
                      →  POST {MAPLE_API_BASE_URL}/internal/slack/workspaces/{teamId}/usage
                         Authorization: Bearer maple_svc_{MAPLE_INTERNAL_SERVICE_TOKEN}
                         { idempotencyKey, inputTokens, outputTokens, model }
                      →  SlackIntegrationService.resolveOrgForTeam(teamId)
                      →  trackTokenUsage({ orgId, …, source: "slack-agent" })  →  Autumn

Where usage is captured. eve exposes provider-reported token counts on exactly one hook event: step.completed, one per model call, carrying usage?: { inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens, costUsd }. turn.completed carries only { sequence, turnId }, and HookContext has no usage accounting (eve's running totals live in a private harness module with no public export). workers-ai-provider populates AI SDK v7 usage in both doGenerate and doStream, so the counts are real.

Auth. Reuses the dedicated SLACK_INTERNAL_SERVICE_TOKEN bearer that already guards resolve/revoke — deliberately not the shared INTERNAL_SERVICE_TOKEN, matching the existing endpoints. Autumn's credential stays on the API side: AUTUMN_SECRET_KEY is org-agnostic, so a bot holding it could write usage against any customer. The bot only ever proves it is the bot; the API decides whose spend it is.

Per step, not per turn. Per-turn batching means mutable state keyed by turn id, which leaks on a turn that never terminates and drops the whole turn's spend if the process dies mid-turn. <sessionId>:<turnId>:<stepIndex> is naturally unique, so it doubles as the idempotency key and a replayed durable step de-duplicates itself. The API namespaces it by team (slack:<teamId>:<key>) so two workspaces can never collide, and trackTokenUsage appends :slack-agent:input|output per its existing convention.

Nothing here can cost a user their reply. trackStepUsage never rejects and is fired un-awaited from the hook; the endpoint answers 202 even when Autumn 500s (the Slack reply already went out — a 5xx would only buy a retry of something Autumn may already have recorded); failures are logged as structured, queryable records via emitAgentLog / Effect.logWarning.

Least privilege on lookup. New resolveOrgForTeam selects only the org column, rather than reusing resolveForBot — attributing a token count must not decrypt (or risk logging) the workspace's bot token and full-access Maple API key.

Skipped, deliberately: zero-token steps (workers-ai-provider reports all-zeros rather than undefined when a stream is truncated before the usage chunk arrives, so these are expected noise), sessions with no Slack team (local dev — no org to bill), and the self-hosted default org (existing trackTokenUsage behaviour).

New env vars / secrets

None. The bot side reuses MAPLE_API_BASE_URL + MAPLE_INTERNAL_SERVICE_TOKEN, which it already needs for workspace resolution; the API side reuses SLACK_INTERNAL_SERVICE_TOKEN, AUTUMN_SECRET_KEY and AUTUMN_API_URL. No migration — usage goes straight to Autumn, as with the existing triage call sites.

Behaviour if unconfigured is the pre-existing shape: without AUTUMN_SECRET_KEY the API accepts and no-ops; without SLACK_INTERNAL_SERVICE_TOKEN the endpoint 401s.

Testing

• apps/api — 8 new route tests in slack-integration.http.test.ts (PGlite + a stubbed Autumn endpoint): org attribution and exact idempotency keys, zero-token side skipped, 202-on-Autumn-failure, default-org exemption, 404 on unknown/revoked team, bearer enforcement (including that the shared internal token is rejected), 401 when unconfigured, 400 on 7 malformed bodies. Full suite 1204 passed | 5 skipped.
• apps/slack-agent — new token-usage.test.ts (report shape, model stamping, skip cases, count coercion, failure swallowing) + 3 new reportTokenUsage cases in maple.test.ts. 44 pass across both files.
• bun typecheck — 33/33 turbo tasks green; apps/slack-agent tsc clean (it is outside the root workspace, so it is typechecked separately).
• Pre-existing failures, unrelated and reproduced on untouched main: apps/web alerts/timezone suites (SchemaError: Missing key at ["environments"]), and the apps/slack-agent Cannot find module 'eve/tools' resolution errors.

Follow-up, not in scope

Subagent spend is reported separately by eve (on action.result with kind: "subagent-result") and is not captured here — this agent defines no subagents today, so there is nothing to miss yet. Cache read/write tokens are carried by eve but dropped, since Autumn only meters ai_input_tokens / ai_output_tokens.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Review fixes

Two adversarial reviews found six confirmed issues on this path; all are fixed in 654f360b.

HIGH — the idempotency key was not unique per model call → silent under-billing.
<sessionId>:<turnId>:<stepIndex> looked unique but is not. eve derives turnId as
turn_${sequence} and advances sequence only in emitTurnEpilogue / emitRecoverableFailedTurn.
A terminal model failure (provider 400, context-length, model-config error) runs
emitFailedStep, which advances nothing — so when the user re-asks, the next turn is minted with
the same turn_${sequence} and stepIndex back at 0, and that retry's step 0 (usually the
largest input step) reuses a key Autumn has already seen. Autumn drops it silently: no log, no 4xx,
just lost revenue. eve's park-and-resume branches (setPendingRuntimeActionBatch,
setPendingAuthorization) return emission state unchanged and have the same shape.
The key now keeps the readable <sessionId>:<turnId>:<stepIndex> prefix and appends a per-process
boot id + monotonic counter. Trade-off, taken deliberately and documented in the source: a
genuine duplicate delivery of one call would now be billed twice — but the bot issues a single
fetch with no retry, so that is near-impossible, whereas the collision above is a live path.

MEDIUM — no upper bound on a financial write. Token counts were validated as any non-negative
int, on a shared service bearer that works against every org, on a plain HttpRouter with no rate
limiting. One POST of 1e15 would land on a customer's Autumn meter and blow their plan limit and
invoice. Capped at 8x the bot's 262,144-token context window; over-cap → 400.

MEDIUM — a persistent Autumn outage was invisible. postTrack swallowed every failure into
console.warn, so the route's Effect.catchCause was dead code: every request answered
202 {"tracked": true} with an Ok span even when nothing was billed. A rotated AUTUMN_SECRET_KEY
— bound with optionalSecret(), which silently omits a key missing from an Infisical environment
with no CI error — could unbill every Slack turn for a week with nobody paged. trackTokenUsage
now returns an outcome; the route emits a structured logError, annotates the span
(maple.billing.*), and the body reports { tracked, outcome } instead of asserting success.
Still 202, still never blocking or retried — the Slack reply has already gone out. The triage
caller ignores the additive return value.

MEDIUM — the billing-failure test asserted nothing. Because of the above, the 500 stub was
swallowed and the test would have passed with the entire error branch deleted. It now asserts both
writes were attempted and that the response no longer claims success, plus a new case covering the
unconfigured-credential (no-credentials) path.

LOW — ops signal mislabeled. logAccess hardcoded "Slack internal resolve …", so usage
requests (including 401s on the usage route) logged as resolve traffic. Lines are now labeled by
the route that produced them.

LOW — no timeout on the Autumn call, which handleUsage awaits inline. At one request per model
step, a stalled Autumn tied up a Worker request per step. Added a 5s AbortSignal.timeout.

LOW — silent zero-token skips. Zero-token steps are skipped as truncation noise; if
workers-ai-provider never surfaces usage for the configured model, that means billing exactly
nothing with no error anywhere. Skips now emit a throttled usage_report_zero_tokens warning
(first occurrence, then at most one per 5 min with a cumulative count). Zeros are still not billed.

Verified correct by both reviews and left alone: the auth (constant-time, maple_svc_ prefix,
fails closed, no fallback to the shared token), per-step usage being incremental not cumulative
(so there is no quadratic over-billing), cached/reasoning tokens already being included, the
un-awaited hook firing, and env plumbing.

bun typecheck green (33/33; the pre-existing alchemy.run.ts(148) Worker type error also exists
on main). bun run test green except the failures already present on main. slack-agent:
typecheck clean, 189/189.

Known gaps (out of scope here — flagging for the record)

1. Compaction spend is never billed. eve's harness/compaction.js calls generateText
   directly and emits no step event and no usage, so context-compaction tokens are invisible to
   this path. Not fixable without upstream eve support.
2. Failed / retried steps burn real tokens but emit no step.completed, so their spend is
   never reported either.
3. Unconfirmed: whether Cloudflare's SSE actually returns usage chunks for the configured model.
   workers-ai-provider calls /ai/run/{model} with {stream:true} and never sets
   stream_options.include_usage. If no chunk carries usage, every step reports all-zeros and this
   feature bills nothing. Needs one real staging Slack turn to confirm — the new
   usage_report_zero_tokens warning (fix 7 above) is the detector for exactly this.

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot ｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[github-actions]

🍁 Maple PR preview

Warning

The latest preview deployment did not complete. The Alchemy deploy outcome was failure; any earlier preview should be treated as stale.

Commit 654f360 · View workflow run

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot ｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}