feat(audio): provider-neutral speech-to-text seam

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 All notable changes to GopherAgent are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/); versions follow [Semantic Versioning](https://semver.org/) — pre-1.0, breaking API changes only require a minor bump.
 
+## [v0.41.0] — 2026-08-10
+
+### Added
+
+- **`pkg/audio` — a provider-neutral speech-to-text seam, so an audio-fed agent is not limited to the one vendor that accepts audio in a message.** Of the three providers here, only Gemini takes audio in a chat message: Anthropic has no audio content block at all, and the OpenAI Chat Completions client cannot express one. Transcribing *before* the message rather than inside it makes the capability portable — every provider can drive an audio-fed agent, because what reaches the model is ordinary text. It is also the cheaper shape for anything long. History is re-sent on every LLM call in a session, so audio carried as a message part is re-uploaded on every subsequent turn: a one-hour recording transcribed once costs one upload, while the same recording as message parts is billed again on each turn of the conversation about it. The package is a stdlib-only leaf holding the `Transcriber` interface plus `Clip`, `Transcript`, `Segment`, and `Options`. Three sentinels — `ErrNoAudio`, `ErrUnsupportedFormat`, `ErrTooLarge` — are separate rather than one error because a live-capture pipeline responds to each differently: an oversized clip must be re-cut into shorter chunks, an unsupported one re-encoded, and an empty one is a caller bug; routing on `errors.Is` beats matching message text. `Transcript.Segments` is nil when the backend emits no timing, which is a normal result rather than a failure, so callers needing timestamps must check rather than assume. `Ext` strips MIME parameters and matches case-insensitively, because browsers' `MediaRecorder` reports `audio/webm;codecs=opus` and some emit the `video/webm` spelling for an audio-only recording — rejecting either would fail a clip the backend decodes fine. (`pkg/audio`)
+- **`openai.NewTranscriber` and `gemini.NewTranscriber`.** The OpenAI implementation drives the audio transcription endpoint and populates `Segments`, `Language`, and `Duration`. It selects the `verbose_json` response format only for whisper models: the `gpt-4o-transcribe` family rejects that format outright rather than degrading, so asking for it everywhere would fail every request instead of merely losing timings. The match is a substring rather than a prefix, since compatible endpoints name the same weights differently. Oversized clips are rejected against the endpoint's 25 MB limit before the upload rather than after transferring the payload. `WithBaseURL` works here as on every other client in the package, so a self-hosted transcription server is a supported target. The Gemini implementation has no dedicated transcription endpoint to call, so it constrains the generation API with a system instruction — without one the model opens with a preamble or summarizes instead of transcribing, and both corrupt a transcript appended verbatim. Its `Segments` is always nil, stated on the type rather than discovered at run time, and its language and vocabulary hints travel as instruction text because the API has no parameter for either. (`pkg/llm/openai/transcriber.go`, `pkg/llm/gemini/transcriber.go`)
+
+### Changed (breaking)
+
+- **A message carrying a media part the adapter cannot render now fails with `agent.ErrUnrenderablePart` instead of being silently dropped.** All three adapters converted `history.MediaPart` with a `switch` that fell through for anything unexpected, and one of them documented the omission as deliberate on the grounds that erroring would force callers to pre-validate. That trade was wrong in the direction that matters. Dropping the part does not degrade the call, it silently changes what the question was: the model receives the caption alone and answers it fluently, and nothing distinguishes that from success — not the response, not the logs, not a schema check, because a well-formed answer is exactly what success looks like. A judge that cannot see the image it is judging is not a degraded judge, it is a random one. Four shapes now fail: an unknown part type, an image with neither `URL` nor `Data`, a parts slice that yields no content at all, and media parts on a non-`user` role, which every adapter previously ignored wholesale — OpenAI behind an explicit role guard, Anthropic and Gemini by rendering media only under their `user` branch. Empty text parts are still skipped, as they carry nothing to lose. `blocksFromMediaParts` and `partsFromMediaParts` grew an `error` return; both are unexported and every call site is inside `GenerateStream`, which already returned one. `isRetryable` treats the sentinel as terminal — the same bytes fail identically on every attempt, so retrying only burns the budget. Callers that relied on a malformed part being ignored will now see the call fail; that is the point. (`pkg/agent/errors.go`, `pkg/agent/retry.go`, `pkg/llm/anthropic`, `pkg/llm/openai`, `pkg/llm/gemini`)
+
+### Fixed
+
+- **The Gemini transcriber reads `FinishReason` before testing for nil content.** A candidate stopped by a content filter arrives with a non-`STOP` reason *and* nil content, so checking content first reported a blocked recording as an empty transcript — a filtered meeting became indistinguishable from a silent one, with no error to act on. The same ordering guards truncation, where returning the accumulated prefix would present half a transcript as the whole. A response with no candidate at all is now an error rather than an empty transcript, since it signals a prompt-level block rather than audio without speech. (`pkg/llm/gemini/transcriber.go`)
+
 ## [v0.40.0] — 2026-08-09
 
 ### Added
@@ -620,6 +635,7 @@ Multi-user, long-running, audit-friendly chat surface — the foundation for sid
 - README section on the permission flow — documents `RequiresConfirmation` × `ConfirmHITL` × `Permissions` interaction.
 - Enum struct tag support in `tools.SchemaFor[T]()` — emit values into JSON-Schema's `enum` array so providers reject invalid values upstream.
 
+[v0.41.0]: https://github.com/hung12ct/gopheragent/releases/tag/v0.41.0
 [v0.40.0]: https://github.com/hung12ct/gopheragent/releases/tag/v0.40.0
 [v0.39.0]: https://github.com/hung12ct/gopheragent/releases/tag/v0.39.0
 [v0.38.0]: https://github.com/hung12ct/gopheragent/releases/tag/v0.38.0

README.md

@@ -60,6 +60,9 @@ go get github.com/hung12ct/gopheragent
   chain for logging, timing, rate limiting, and tracing.
 - **Multi-provider** — OpenAI, Anthropic, Gemini, Vertex, OpenRouter, and OpenAI-compatible
   backends, each in its own subpackage; multi-model routing; sampling controls.
+- **Speech to text** — `pkg/audio` transcribes with Whisper or Gemini behind one
+  interface, ahead of the loop, so any provider can drive an audio-fed agent and
+  a long recording is uploaded once rather than re-sent every turn.
 - **Sub-agents & async** — sub-agent streaming, conversation forking, background
   workers, first-class task tracking.
 - **Cross-session memory** — a post-session consolidator distills transcripts

pkg/agent/errors.go

@@ -93,6 +93,22 @@ var (
 	// caller must change the request or surface the block to the user.
 	ErrLLMContentBlocked = errors.New("agent: LLM stopped generating for a content policy")
 
+	// ErrUnrenderablePart is returned by a provider adapter when a message
+	// carries a history.MediaPart the adapter cannot put on the wire — an
+	// unsupported part type, or a part whose payload is missing.
+	//
+	// Failing is the point. The alternative, dropping the part and sending
+	// the rest, produces a fluent well-formed answer from a model that never
+	// received the media, and nothing distinguishes that from success: not
+	// the response, not the logs, not a schema check. A judge that cannot see
+	// the image it is judging is not a degraded judge, it is a random one.
+	//
+	// Deterministic like ErrLLMAuth, not transient like ErrLLMFailure — the
+	// same message fails identically on every retry. isRetryable treats it as
+	// terminal; the caller must re-encode the part or choose a provider that
+	// supports it.
+	ErrUnrenderablePart = errors.New("agent: message carries a media part this provider cannot render")
+
 	// ErrContextCancelled is returned when the request context is cancelled mid-loop.
 	ErrContextCancelled = errors.New("agent: operation cancelled")
 

pkg/agent/retry.go

@@ -57,13 +57,15 @@ func (r *RetryConfig) delay(attempt int) time.Duration {
 }
 
 // isRetryable returns false for errors that fail identically on every
-// attempt: context-level cancellation, and a provider content-policy stop
-// (deterministic for a given prompt — retrying only burns the budget).
+// attempt: context-level cancellation, a provider content-policy stop
+// (deterministic for a given prompt — retrying only burns the budget), and a
+// message the adapter cannot render (the same bytes fail the same way).
 func isRetryable(err error) bool {
 	if err == nil {
 		return false
 	}
 	return !errors.Is(err, context.Canceled) &&
 		!errors.Is(err, context.DeadlineExceeded) &&
-		!errors.Is(err, ErrLLMContentBlocked)
+		!errors.Is(err, ErrLLMContentBlocked) &&
+		!errors.Is(err, ErrUnrenderablePart)
 }

pkg/audio/audio.go

@@ -0,0 +1,170 @@
+// Package audio defines provider-neutral types for speech-to-text.
+//
+// Transcription is deliberately kept outside the agent loop. Of the three
+// LLM providers this framework ships, only Gemini accepts audio in a chat
+// message today; Anthropic has no audio content block at all, and the OpenAI
+// Chat Completions client cannot express one. Routing audio through a
+// Transcriber first turns it into ordinary text, so every provider can drive
+// an audio-fed agent — and the transcript, not the waveform, is what lands in
+// session history.
+//
+// That last point matters for cost as much as for portability. History is
+// re-sent on every LLM call in a session, so inline audio in a message would
+// be re-uploaded on every subsequent turn. A one-hour meeting transcribed once
+// costs one upload; the same meeting carried as message parts would be billed
+// again on each turn of the conversation about it.
+package audio
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// Sentinel errors returned by Transcriber implementations. They are separate
+// types because they demand opposite responses: an oversized clip should be
+// split and retried, an unsupported format should be re-encoded, and an empty
+// clip is a caller bug. Route on these with errors.Is rather than matching on
+// message text.
+var (
+	// ErrNoAudio is returned when a clip carries no sample data.
+	ErrNoAudio = errors.New("audio: clip has no data")
+
+	// ErrUnsupportedFormat is returned when the clip's MIME type is not one
+	// the backend accepts. Re-encode and retry; retrying as-is always fails.
+	ErrUnsupportedFormat = errors.New("audio: unsupported format")
+
+	// ErrTooLarge is returned when a clip exceeds the backend's per-request
+	// size limit. Callers feeding a live stream should respond by cutting
+	// shorter chunks, not by retrying the same clip.
+	ErrTooLarge = errors.New("audio: clip exceeds provider size limit")
+)
+
+// Clip is a single piece of audio to transcribe.
+//
+// Data holds the whole clip in memory. That is deliberate rather than a
+// missing optimization: both backends buffer the full payload anyway — the
+// OpenAI client assembles its multipart body into a bytes.Buffer, and Gemini
+// inline data is a []byte field — so an io.Reader here would add a streaming
+// API that does not stream. Callers transcribing long recordings should cut
+// them into chunks, which is what a live-capture pipeline does regardless.
+type Clip struct {
+	// MIME is the IANA media type, e.g. "audio/wav" or "audio/webm".
+	// Parameters are allowed and ignored: browsers' MediaRecorder reports
+	// "audio/webm;codecs=opus", and that value is accepted verbatim.
+	MIME string
+
+	// Data is the raw encoded clip — the bytes of a .wav or .webm file, not
+	// decoded PCM samples.
+	Data []byte
+}
+
+// Options tunes a single transcription request. The zero value is valid and
+// asks the backend to auto-detect everything.
+type Options struct {
+	// Language is an ISO-639-1 hint such as "en" or "vi". Empty means
+	// auto-detect. Setting it cuts latency and materially improves accuracy
+	// on short clips, where there is little signal to detect from.
+	Language string
+
+	// Prompt biases decoding toward expected vocabulary — proper nouns,
+	// product names, jargon, acronyms. For a chunked live stream, passing the
+	// tail of the previous chunk's text carries context across the seam and
+	// reduces mid-word splits.
+	Prompt string
+}
+
+// Segment is a timed span of transcribed speech.
+type Segment struct {
+	Start time.Duration // offset from the start of the clip
+	End   time.Duration
+	Text  string
+}
+
+// Transcript is the result of transcribing one clip.
+type Transcript struct {
+	// Text is the full transcription. Always populated on success.
+	Text string
+
+	// Language is the detected or configured language, best-effort, and may
+	// be empty when the backend reports none.
+	//
+	// The spelling is the backend's, not a normalized code: Whisper returns
+	// an English name ("english"), while a backend given Options.Language
+	// echoes that ISO-639-1 code back. Compare it against a fixed set at your
+	// own risk; it is for display and logging.
+	Language string
+
+	// Duration is the length of the source audio, best-effort. Zero when the
+	// backend does not report it.
+	Duration time.Duration
+
+	// Segments carries per-span timings when the backend provides them.
+	// Nil is a normal result, not an error: some backends transcribe without
+	// emitting any timing at all. Callers that need timestamps must check for
+	// nil rather than assuming a populated slice.
+	Segments []Segment
+}
+
+// Transcriber converts audio into text. Implementations live in the provider
+// subpackages under pkg/llm.
+//
+// Implementations must be safe for concurrent use: a live-capture pipeline
+// transcribes overlapping chunks from several goroutines to keep up with
+// real time, which is the primary use for this seam.
+type Transcriber interface {
+	Transcribe(ctx context.Context, clip Clip, opts Options) (Transcript, error)
+}
+
+// extByMIME maps accepted media types to the file extension backends use to
+// infer the container format. Both the audio/* and video/* spellings of the
+// shared containers are listed: MediaRecorder emits "video/webm" for an
+// audio-only recording on some browsers, and rejecting that would fail a clip
+// the backend decodes fine.
+var extByMIME = map[string]string{
+	"audio/wav":       "wav",
+	"audio/x-wav":     "wav",
+	"audio/wave":      "wav",
+	"audio/vnd.wave":  "wav",
+	"audio/mpeg":      "mp3",
+	"audio/mp3":       "mp3",
+	"audio/mp4":       "m4a",
+	"audio/x-m4a":     "m4a",
+	"audio/webm":      "webm",
+	"video/webm":      "webm",
+	"video/mp4":       "mp4",
+	"audio/ogg":       "ogg",
+	"application/ogg": "ogg",
+	"audio/opus":      "ogg",
+	"audio/flac":      "flac",
+	"audio/x-flac":    "flac",
+}
+
+// Ext returns the file extension for a media type, or "" when the type is not
+// a recognized audio container. Parameters after ";" are stripped and the type
+// is matched case-insensitively, so "AUDIO/WEBM;codecs=opus" resolves to
+// "webm".
+func Ext(mime string) string {
+	base, _, _ := strings.Cut(mime, ";")
+	return extByMIME[strings.ToLower(strings.TrimSpace(base))]
+}
+
+// Validate reports whether the clip is well-formed enough to send. It does not
+// enforce per-provider size limits — those belong to the implementation that
+// knows them.
+func (c Clip) Validate() error {
+	if len(c.Data) == 0 {
+		return ErrNoAudio
+	}
+	// The sentinels already carry the package prefix, so wrapping adds only
+	// the detail — otherwise every message reads "audio: audio: ...".
+	if c.MIME == "" {
+		return fmt.Errorf("%w: MIME is required", ErrUnsupportedFormat)
+	}
+	if Ext(c.MIME) == "" {
+		return fmt.Errorf("%w: %q", ErrUnsupportedFormat, c.MIME)
+	}
+	return nil
+}

pkg/audio/audio_test.go

@@ -0,0 +1,77 @@
+package audio
+
+import (
+	"errors"
+	"strings"
+	"testing"
+)
+
+func TestExtStripsParametersAndCase(t *testing.T) {
+	// MediaRecorder reports the codec as a parameter, and some browsers use
+	// the video/* spelling for an audio-only recording. Both must resolve.
+	for _, tc := range []struct{ mime, want string }{
+		{"audio/webm", "webm"},
+		{"audio/webm;codecs=opus", "webm"},
+		{"audio/webm; codecs=opus", "webm"},
+		{"video/webm;codecs=opus", "webm"},
+		{"AUDIO/WEBM;CODECS=OPUS", "webm"},
+		{"  audio/wav  ", "wav"},
+		{"audio/mpeg", "mp3"},
+		{"audio/x-m4a", "m4a"},
+		{"audio/flac", "flac"},
+		{"audio/opus", "ogg"},
+		{"text/plain", ""},
+		{"", ""},
+	} {
+		if got := Ext(tc.mime); got != tc.want {
+			t.Fatalf("Ext(%q) = %q, want %q", tc.mime, got, tc.want)
+		}
+	}
+}
+
+func TestClipValidate(t *testing.T) {
+	data := []byte{0x1, 0x2}
+	for _, tc := range []struct {
+		name string
+		clip Clip
+		want error
+	}{
+		{"ok", Clip{MIME: "audio/wav", Data: data}, nil},
+		{"ok with codec parameter", Clip{MIME: "audio/webm;codecs=opus", Data: data}, nil},
+		{"no data", Clip{MIME: "audio/wav"}, ErrNoAudio},
+		{"no mime", Clip{Data: data}, ErrUnsupportedFormat},
+		{"unknown mime", Clip{MIME: "application/pdf", Data: data}, ErrUnsupportedFormat},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			err := tc.clip.Validate()
+			if tc.want == nil {
+				if err != nil {
+					t.Fatalf("Validate() = %v, want nil", err)
+				}
+				return
+			}
+			if !errors.Is(err, tc.want) {
+				t.Fatalf("Validate() = %v, want %v", err, tc.want)
+			}
+			// The sentinels already carry the package prefix; wrapping must
+			// add only detail, not a second "audio: ".
+			if strings.Contains(err.Error(), "audio: audio:") {
+				t.Fatalf("Validate() = %q, want a single package prefix", err)
+			}
+		})
+	}
+}
+
+// The sentinels exist so a live-capture pipeline can route on them: a
+// too-large clip should be re-cut, an unsupported one re-encoded. Distinct
+// identities are the contract, so guard against a careless collapse into one.
+func TestSentinelsAreDistinct(t *testing.T) {
+	all := []error{ErrNoAudio, ErrUnsupportedFormat, ErrTooLarge}
+	for i, a := range all {
+		for j, b := range all {
+			if i != j && errors.Is(a, b) {
+				t.Fatalf("errors.Is(%v, %v) = true, want distinct sentinels", a, b)
+			}
+		}
+	}
+}

pkg/history/types.go

@@ -72,10 +72,20 @@ type Message struct {
 }
 
 // PartType enumerates the kinds of content a MediaPart can hold.
-// Video and audio are intentionally omitted for now — Anthropic does not
-// accept them in messages today, and Gemini video requires Files-API upload
-// semantics that do not fit the simple inline model. Video remains on the
-// tool path via media_analyze.
+//
+// Video and audio are intentionally omitted. Anthropic accepts neither in a
+// message, and Gemini video requires Files-API upload semantics that do not
+// fit the simple inline model. Video remains on the tool path via
+// media_analyze; audio goes through pkg/audio, whose Transcriber converts it
+// to text before it reaches a message.
+//
+// Transcribing rather than embedding is also the cheaper shape for anything
+// long. History is re-sent on every LLM call in a session, so audio carried
+// as a message part is re-uploaded on every subsequent turn — a one-hour
+// recording would be billed again on each turn of the conversation about it.
+//
+// A part whose type an adapter cannot render fails the request with
+// agent.ErrUnrenderablePart rather than being dropped.
 type PartType string
 
 const (