Skip to content

feat(binding-llm): server framing decode and forward - #2569

Open
jfallows wants to merge 13 commits into
developfrom
claude/stoic-sagan-9d81tf
Open

feat(binding-llm): server framing decode and forward#2569
jfallows wants to merge 13 commits into
developfrom
claude/stoic-sagan-9d81tf

Conversation

@jfallows

@jfallows jfallows commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Description

LlmServerFactory — the llm server stream factory. On BEGIN, resolves the inbound dialect via LlmDialectResolver (from the :path/headers of the accepted HttpBeginExFW), selects the matching LlmContentDecoder from LlmContentDecoderFactory keyed by the dialect's content-type, and forwards decoded canonical DATA/FLUSH frames to the routed application binding, with LlmBeginEx.dialect annotated on the forwarded BEGIN.

  • Framing decode runs unconditionally on every route, same-dialect included, per the design's layering rule — no dialect translation happens here (that's llm client's job when dialects differ, tracked separately).
  • When no decoder is registered for the resolved content-type (e.g. non-streaming JSON, for which no LlmContentDecoderSpi implementation exists yet), the body is forwarded opaquely as DATA followed by a terminal raw FLUSH on END, rather than guessing at an undefined decode contract.
  • The reply direction (application → network) is a generic, unmodified byte passthrough — no re-encoding logic is implemented here, since that isn't this issue's scope either.
  • An unroutable or unresolved-dialect request is rejected by returning null from newStream, relying on the engine's existing auto-reset behavior rather than a bespoke reject handler.
  • Modelled on binding-mcp's server-side stream handling patterns (buffer-slot decode/resume, *State bitmask, generic do* frame builders).

Covered by new k3po specs (sse.passthrough, opaque.fallback) under streams/network/streams/application, each with paired runtime IT (LlmServerIT, against a live engine) and peer-to-peer IT (NetworkIT/ApplicationIT, scripts run directly against each other) coverage, plus a unit test for the new k3po EL helper functions (LlmFunctionsTest).

Fixes #2484

This branch stacks on #2568 (dialect detection dispatcher and fixed-dialect config, #2483), which itself stacks on the earlier M1 protocol-scaffold PRs — so this diff includes those commits until they merge to develop, at which point this PR's diff will shrink to just this framing-decode commit.

Generated by Claude Code

@claude
feat(binding-llm.spec): define llm.idl stream extension types
cb31300 
Scaffold incubator/binding-llm.spec per AGENTS.md conventions and define
LlmBeginEx, LlmDataEx, and the LlmFlushEx union, modelled on
binding-mcp.spec's idl.

LlmBeginEx carries dialect only; model routing is deferred. LlmDataEx has
no fields: content flows through the DATA frame's own payload octets and
INIT/FIN through its existing flags, so nothing survives in the extension
once block identity moves to the FLUSH plane. LlmFlushEx is a 7-case union
covering message start, block start/end, finish, usage, keepalive, and an
opaque native/raw case for re-encoding events a same-dialect route doesn't
recognize.

Fixes #2476

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AuZoMsETwEczJx3cbkb8EJ
@claude
feat(binding-llm): module scaffold with Incubating LlmBindingInfo
daf8a87 
Scaffolds incubator/binding-llm and incubator/binding-llm.conf, modelled
on binding-mcp's SERVER/CLIENT BindingContext structure. LlmBindingInfo
is annotated @Incubating so type: llm config loading is gated behind
ZILLA_INCUBATOR_ENABLED via FeatureFilter, matching the AmqpBindingInfo/
PgsqlBindingInfo/RisingwaveBindingInfo precedent.

Fixes #2477

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0142buJWS7C89AKr9uDJtSy4
@claude
feat(binding-llm): LlmContentDecoderSpi internal SPI, keyed by conten…
b822228 
…t-type

Registers by content-type and hands back a per-stream LlmContentDecoder;
stays in an internal, unexported package for now with no concrete
implementation registered yet.

Fixes #2478

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH
@jfallows jfallows mentioned this pull request Sep 11, 2026
Open
@claude
feat(binding-llm): application/json content decoder
8faad28 
Closes the non-streaming half of #2478's own scope: "Non-streaming
application/json goes through the same abstraction as one event, rather
than a special-cased branch." Only text/event-stream had an
LlmContentDecoderSpi implementation; application/json requests
(non-streaming dialect responses) had no decoder to dispatch to.

LlmJsonContentDecoder treats the entire buffered document as a single
event (one data + one flush call, no framing loop), mirroring
LlmSseContentDecoder's structure and unit-test conventions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH
@claude
feat(binding-llm): SSE content decoder for text/event-stream
b30e1f4 
Implements LlmContentDecoderSpi for text/event-stream, decoding blank-line-
delimited SSE framing (data:/event:/id: fields, comment lines, CRLF/CR/LF
line endings) into event DATA + FLUSH frames per the WHATWG SSE parsing
algorithm. Framing decode only, independent of any dialect-specific (e.g.
OpenAI, Anthropic) payload interpretation.

The last-event-id persists across dispatches until overwritten by a
subsequent id: field, matching SSE reconnection semantics, and is reported
as the flush boundary's associated bytes. A trailing lone CR at the edge of
the currently available bytes is deliberately left unconsumed (returned as
no progress) since it may still turn out to be part of a CRLF pair once
more bytes arrive - avoiding a premature, ambiguous line-terminator decision
on a streamed input.

Representational-slack note for the round-trip decision this issue flags:
this decoder normalizes field ordering and does not preserve field-value
whitespace beyond the single optional space after the colon that the SSE
spec itself strips, so re-encoding from the decoded data/flush frames alone
cannot byte-exactly reproduce arbitrary source framing - only semantic
equality should be asserted against this decoder's output.

Fixes #2479

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kK9z4ywuPJe33jm6W1fik
@claude
feat(binding-llm): LlmDialectFactorySpi / LlmDialect exported SPI
c543776 
Defines the pluggable-dialect contract for binding-llm, exported from the
start (unlike LlmContentDecoderSpi, which stays internal): LlmDialect
exposes name()/detect()/contentType() plus supplyDecoder(Kind)/
supplyEncoder(Kind) returning common-json JsonTransform stages, and
LlmDialectFactorySpi is the ServiceLoader-registered entry point.
HttpHeaders is a minimal read-only accessor for detect(path, headers),
since no HTTP header abstraction previously existed in this codebase and
pulling in jakarta.ws.rs would add a dependency never otherwise used here.
Kind is nested on LlmDialect, distinguishing request/response schemas.

No concrete dialect implementations yet (OpenAI/Anthropic land later) --
module-info.java exports the dialect package and declares uses without a
corresponding provides. Unit-tested via a stub LlmTestDialect/
LlmTestDialectFactorySpi registered under test-scope META-INF/services,
mirroring this module's existing LlmContentDecoderSpi/
LlmTestContentDecoderFactorySpi pattern.

Fixes #2480

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6
@jfallows
jfallows force-pushed the claude/stoic-sagan-9d81tf branch from b8458bb to e4f28d0 Compare September 12, 2026 02:08
@claude
feat(binding-llm): resolve LlmDialect content-type per request, not p…
dc5f206 
…er instance

contentType() previously took no parameters, so a dialect could only
report one fixed content-type for its lifetime -- insufficient for an
API whose response framing (event-stream vs. a single JSON document)
depends on a flag in the request body, since neither contentType() nor
detect(String, HttpHeaders) offered any way to inspect it.

Adds HttpRequestBody, a minimal read-only scalar-member accessor
mirroring HttpHeaders, and changes contentType() to
contentType(Kind, HttpHeaders, HttpRequestBody): Kind lets request and
response resolve independently (a dialect's request body content-type
can be fixed while its response varies), and the headers/body context
lets that resolution depend on the actual request rather than being
fixed at dialect-instance-creation time. Both parameters are nullable
for callers without that context available.

LlmTestDialect now resolves text/test-event-stream for a streaming
response and application/test+json otherwise, exercising the new
per-Kind, per-request resolution the stub previously couldn't express.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6
@claude
feat(binding-llm.spec): canonical event vocabulary — choice index, to…
037b0ce 
…ken detail

Extend LlmFlushEx's block-lifecycle skeleton (from #2476) with the two
places OpenAI's parallel completions and per-token detail need to survive
in the vocabulary, per the issue's scope:

- choiceIndex (default 0) on messageStart, blockStart, blockEnd, finish,
  and native/raw: the (choice, block) compound index's outer half.
  Anthropic is always choice 0, so every existing dialect mapping is
  unaffected; OpenAI's n > 1 becomes one messageStart per parallel
  completion, distinguished by choiceIndex. usage stays choiceIndex-free
  since every dialect reports it aggregated across choices, never per
  choice.
- logProbability (nullable) on LlmDataEx: the one per-delta detail the
  vocabulary carries directly, for dialects exposing per-token detail
  (OpenAI logprobs) without reopening the DATA/FLUSH split from #2476 or
  growing the vocabulary for the full log-probability structure — richer
  detail than one value per token stays behind LlmNativeFlushEx.

Documents the three lossiness cases as doc comments alongside the fields
they concern: choiceIndex and logProbability both drop out on any
cross-dialect route to Anthropic (structurally exactly one choice, no
per-token detail); message-start input token counts are resolved by
decoupling inputTokens into its own deferred usage event rather than
emitting a placeholder on messageStart and correcting it later, so a
source that discloses tokens late (OpenAI) just emits usage late instead
of needing a correction event.

Fixes #2481

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AYGZDrytLjmG2AQqUN1hsu
@claude
feat(binding-llm): event sequence mapper for anthropic and openai dia…
0ee8b63 
…lects

Translates between each dialect's native streaming event sequence and the
canonical vocabulary from #2557, in both directions:

- LlmAnthropicEventMapper: holds input_tokens from message_start until the
  paired usage event at message_delta; tracks the currently open block's
  type to know when content_block_stop needs a canonical blockEnd (tool
  calls only) and to route content_block_delta payloads (text_delta vs
  input_json_delta) on encode.
- LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into
  the canonical (Anthropic-shaped) block index via a per-stream map, and
  synthesizes blockEnd lazily -- deferred until the next tool call starts
  or the stream finishes, since OpenAI has no explicit block-close event.

Unit-tested against both worked-example tables from the issue (message
role/content/tool-call cardinality changes in each direction), plus the
held-usage/already-consumed and lazy-blockEnd edge cases.

Fixes #2482

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw
@claude
feat(binding-llm): dialect detection dispatcher and fixed-dialect config
db57685 
LlmDialectResolver dispatches path/header detection across every LlmDialect
registered via LlmDialectFactorySpi, without hardcoding any dialect's
signals. A configured fixed dialect name bypasses detection entirely,
including when it matches no registered dialect. When detection matches
more than one dialect, or none, resolution is ambiguous and returns null
so the caller rejects the request rather than guessing.

LlmOptionsConfig adds the optional server-kind `dialect` option (schema,
config, and adapter) used to pin a fixed dialect.

Fixes #2483

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AtkteS7C66qiVLGeEDJ2FP
@claude
feat(binding-llm): server framing decode and forward, annotate LlmBeg…
04127e0 
…inEx.dialect

Wire llm server (LlmServerFactory) to resolve the inbound dialect via
LlmDialectResolver, decode framing through the matching LlmContentDecoderSpi
(SSE today), and forward canonical DATA/FLUSH frames to the routed
application binding with LlmBeginEx.dialect annotated. When no decoder is
registered for the dialect's content-type, the body is forwarded opaquely as
DATA followed by a terminal raw FLUSH. The reply direction is a generic,
unmodified byte passthrough back to the network. Unroutable or
unresolved-dialect requests are rejected by returning null from newStream,
relying on the engine's existing auto-reset behavior.

Covered by new k3po specs (sse.passthrough, opaque.fallback) with paired
runtime (LlmServerIT) and peer-to-peer (NetworkIT/ApplicationIT) coverage,
plus a LlmFunctionsTest unit test for the new k3po EL helpers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM
@claude
fix(binding-llm): defer non-streaming decoder's terminal flush to end…
f4cb9a5 
… of stream

LlmServerFactory calls a stream's LlmContentDecoder incrementally, once per
network DATA frame, so a fragment-tolerant decoder (SSE) can report event
boundaries as they're recognized. LlmJsonContentDecoder didn't tolerate this:
it treated whatever bytes any single call received as the complete document,
so a JSON body split across more than one DATA frame (arbitrarily likely,
never guaranteed to arrive in one frame) produced a premature terminal flush
per fragment instead of one flush for the whole document.

Fix keeps LlmServerFactory's decode loop unchanged for every decoder: still
called incrementally, fragment by fragment, as bytes arrive (no buffering of
the whole body). LlmContentDecoder now also gets one call with an empty
range after the network stream ends, so a decoder without a self-contained
end-of-document marker can defer its terminal event to that call.
LlmJsonContentDecoder forwards data on every non-empty call and only
reports the terminal event on the empty one.

Covered by a new k3po scenario (json.fragmented) whose network side splits
the JSON body across two DATA frames via a mid-write flush, verified against
a live engine (LlmServerIT), peer-to-peer (NetworkIT/ApplicationIT), and at
the decoder unit level (LlmJsonContentDecoderTest).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM
@claude
fix(binding-llm): adapt to LlmDialect.contentType(Kind, headers, body)
5f11258 
The dialect stack this branch builds on now resolves content-type per
request rather than per dialect instance (contentType() gained Kind,
HttpHeaders, and HttpRequestBody parameters), so a dialect's response
framing can depend on a flag in the request body instead of being fixed
for the dialect's lifetime.

LlmServerFactory resolves content-type for the request direction, using
the headers already extracted for dialect detection and no request-body
context (unavailable at BEGIN time). LlmTestSseDialect/LlmTestJsonDialect
report their fixed content-type regardless of kind/headers/body, matching
their existing single-purpose role in the sse.passthrough/json.fragmented
scenarios.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mHT1vrPATQGSdRcB8NNKM
@jfallows
jfallows force-pushed the claude/stoic-sagan-9d81tf branch from 0510c5d to 5f11258 Compare September 12, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

llm server: framing decode and forward, annotate LlmBeginEx.dialect

2 participants

@jfallows @claude