Skip to content

feat(agent-cache/ai): cache streaming LLM responses via wrapStream#262

Open
amitkojha05 wants to merge 1 commit into
BetterDB-inc:masterfrom
amitkojha05:feat/ai-sdk-stream-caching
Open

feat(agent-cache/ai): cache streaming LLM responses via wrapStream#262
amitkojha05 wants to merge 1 commit into
BetterDB-inc:masterfrom
amitkojha05:feat/ai-sdk-stream-caching

Conversation

@amitkojha05

Copy link
Copy Markdown
Contributor

Summary

The agent-cache Vercel AI SDK adapter currently implements wrapGenerate for non-streaming calls. Streaming responses are not cached, requiring callers to manually accumulate streamed content and invoke store() themselves.

This PR adds support for wrapStream.

On a cache miss, the upstream stream is tee'd:

  • One branch flows to the caller unchanged.
  • The second branch accumulates text-delta chunks and stores the final response via cache.llm.store() on finish.

On a cache hit:

  • doStream is skipped entirely.
  • The cached response is replayed as a single text-delta event followed by a finish event.
  • The finish event includes:
providerMetadata: {
  agentCache: {
    hit: true,
  },
}

Tool-call streams are not cached, matching existing wrapGenerate behavior.

Additional guarantees:

  • Store failures never affect the caller-facing stream.
  • check() failures fall through to the upstream provider.
  • Fully backward compatible; existing wrapGenerate consumers are unaffected.

Changes

adapters/ai.ts

  • Added synthesizeStreamFromCached()
  • Added accumulateAndStore()
  • Added wrapStream to the middleware return object
  • No changes to existing wrapGenerate behavior

adapters/__tests__/ai.test.ts

Added 5 new tests:

  1. Cache miss → cache hit flow
  2. providerMetadata included on cache hits
  3. Tool-call stream bypass
  4. Upstream error propagation
  5. Store failure does not break caller stream

CHANGELOG.md

  • Added v0.7.0 entry

README.md

  • Updated Vercel AI SDK documentation section

Testing

pnpm --filter @betterdb/agent-cache test
pnpm --filter @betterdb/agent-cache build

@amitkojha05 amitkojha05 force-pushed the feat/ai-sdk-stream-caching branch 2 times, most recently from baade71 to 13371cd Compare June 19, 2026 18:42
@amitkojha05 amitkojha05 force-pushed the feat/ai-sdk-stream-caching branch from 13371cd to e938a50 Compare June 19, 2026 18:55
@amitkojha05

Copy link
Copy Markdown
Contributor Author

Hey @KIvanow and @jamby77 ,Please review this PR ;would love to work on your feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant