From c405ad1d4b1da061c11b3d26761252d41dcc62d3 Mon Sep 17 00:00:00 2001 From: Atish Patel Date: Wed, 29 Jul 2026 08:36:09 -0500 Subject: [PATCH 01/32] feat(agent): fix Anthropic prompt caching with Databricks (+ MCP proxy/TLS passthrough) (#3463) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit > On 8 tasks matched by name across the two runs, cost fell $8.36 → $1.77 (4.71×) and wall-clock 12,423 s → 1,085 s (11.45×). ## Summary Two independent, self-contained fixes to `buzz-agent`/`buzz-acp`, split out of the benchmark branch so they can land while the harness work continues: 1. **Request and surface Anthropic prompt caching.** buzz never sent a `cache_control` breakpoint, so on the Databricks Anthropic route `cache_read_input_tokens` was **structurally always 0** and the ~10× cache-read discount was never claimed. This teaches `anthropic_body()` to mark the cacheable prefix, and plumbs the cache split end-to-end so accounting can price it. 2. **Pass proxy + TLS-trust env into MCP tool subprocesses**, so agent tools on a proxy-only host stop reporting a live network as offline. ## Why the caching gap matters The Anthropic Messages API does **not** cache unless the request carries a `cache_control` breakpoint, and the Databricks AI Gateway — a third-party proxy in front of the model, in the same category as Bedrock/Vertex — does **not** auto-cache (only the first-party Anthropic API and Claude-on-AWS do zero-config caching). So every request was billed cold. Measured live against the Databricks gateway (`databricks-claude-opus-5`, 2026-07-28), the same call with and without a single `cache_control` marker: | Run | `input_tokens` | `cache_creation` | `cache_read` | latency | |---|---|---|---|---| | No `cache_control`, two byte-identical calls | 121,625 | 0 | **0** | ~9.3 s | | With one marker — cold (write) | 4 | 121,625 | 0 | 9.3 s | | With one marker — warm (read) | 4 | 0 | **121,625** | **4.5 s** | One marker moved 121,625 tokens from full-price input to a 0.1× cache read and roughly halved latency (a clean, isolated ~2.07× prefill speedup on this single-threaded microbenchmark). The gateway honours `cache_control`; buzz simply never sent it. At fleet scale this was a real budget item. Across matched Terminal-Bench solo sweeps (89 tasks, `-n 20`, before the fix), the two OpenAI-route models independently landed at ~86–87% cache reads — the expected shape for an agentic loop, where system + tools + append-only history repeat every turn — while the Anthropic route returned a hard 0% on every receipt: | Condition | Route | Input tokens | Cache reads | Cost | Cost if uncached | Discount | |---|---|---|---|---|---|---| | luna (`gpt-5-6`) | OpenAI | 20,320,818 | **17.7M (87.0%)** | $6.96 | $22.87 | **3.28×** | | sol (`gpt-5-6`) | OpenAI | 22,312,290 | **19.2M (85.9%)** | $37.07 | $123.35 | **3.33×** | | opus (`claude-opus-5`) | Anthropic | 12,459,822 | **0 (0.0%)** | $81.31 | $81.31 | **1.00×** | Applying luna's measured 87% read rate to the opus token counts at list prices (`input $5/M`, `cached_input $0.5/M`, `output $25/M`) puts the opus run at **~$32.53 vs the $81.31 actually paid — a ~60% overspend on those 49 trials (~$89 on a full sweep)**. That is an upper bound (it prices every cached token at the 0.1× read rate and ignores the 1.25× write premium), and the opus discount is structurally smaller than luna/sol's because opus emits ~3.5× more uncacheable output per trial, which sets a floor on what caching can recover. There is also a plausible **second-order effect**: Databricks appears to meter its per-minute rate limit on *uncached* input tokens, so the missing cache also cost rate-limit headroom — the opus endpoint lost 63% of its trials to fatal 429s while running alone at one-third of a GPT endpoint's raw throughput. This is a hypothesis, not a proven mechanism (the only zero-cache condition is also the only Anthropic endpoint), but it is the reading that explains the throttling with one rule instead of two. ## Post-fix results (provisional — first trials of an in-flight re-run) On 8 tasks matched by name across the two runs, cost fell **$8.36 → $1.77 (4.71×)** and wall-clock **12,423 s → 1,085 s (11.45×)**. | Metric | before (`4a955a858`) | after (`3bef1f6a`) | |---|---|---| | Cache reads as % of input | **0.0%** | **78.7%** (still climbing toward the ~86% steady state) | | `cost_usd_no_cache_discount / cost_usd` | **1.00×** | **2.18×** (tracking the projected ~2.5×) | | Trials with a fatal 429 (same `-n 20`) | **63%** | **15–19%** | To be clear about attribution: **~2× of that is the clean prefill saving from caching itself**; the rest is second-order — cached requests burn far less rate-limit budget, so they stall less and redo less destroyed work. The 11.45× is a system-level result specific to this throttled workspace, not a caching benchmark. Quality held (7/8 solved in each run). A controlled low-`-n` A/B (neither arm hitting a 429), which the `BUZZ_AGENT_PROMPT_CACHING` opt-out exists to enable, is still owed before this becomes a published claim. ## What changed ### 1. Request caching (`llm.rs`, `config.rs`) `anthropic_body()` emits ephemeral `cache_control` breakpoints, gated by `BUZZ_AGENT_PROMPT_CACHING` (**default on**, `=0` to opt out): - **Static prefix** — marker on the `system` block. Prefix order is `tools → system → messages`, so this single marker caches **tools + system** together. Byte-identical on every turn of a run, and survives a context handoff (system/tools come from cfg/mcp, not `self.history`). - **Rolling tail + leapfrog** — marker on the last block of the last **two** messages. The append-only history re-reads the prior turn's prefix from cache; marking two messages (not one) keeps consecutive breakpoints inside Anthropic's **20-block lookback window** even as tool parallelism rises, avoiding a silent full-price miss. An empty system prompt stays a bare string (Anthropic rejects empty text blocks), and below-threshold prefixes are silently not cached, so the flag is safe on by default. ### 2. Surface the cache split end-to-end — the plumbing (`types.rs`, `llm.rs`, `agent.rs`, `lib.rs`, `usage.rs`, `acp.rs`) This is the part that makes gaps like the one above **visible** instead of silent. A consumer that prices all of `input_tokens` at the full rate can't tell a route that's caching from one that isn't — the total looks right either way. So: - `LlmResponse` gains `cached_input_tokens` (a **subset** of `input_tokens`, never an addition); `parse_anthropic` / `parse_openai` / `parse_responses` each populate it. - A `usage_first()` helper reads the cache count wherever a provider hides it — flat `cache_read_input_tokens` (Anthropic), `prompt_tokens_details.cached_tokens` (OpenAI chat), `input_tokens_details.cached_tokens` (Responses) — taking the **first present value, never a sum**. Reading only flat keys is exactly why the OpenAI route's nested `cached_tokens` had *also* been going unclaimed: `prompt_tokens` is already inclusive, so the total looked correct while the discount silently went unreported. - The per-turn/per-session accumulators and the goose `usage_update` payload now carry `accumulatedCachedInputTokens`; `buzz-acp` deserializes it (`serde` default `0` for goose, which doesn't send it) and logs `cached=`. ### 3. Fix a Databricks MLflow-route double-count (`llm.rs`) The Databricks MLflow route reports the flat Anthropic-spelled `cache_read_input_tokens` *alongside* an already-inclusive `prompt_tokens`, so the old code summed them and nearly doubled the count — inflating both the context-budget gate and cost. `openai_chat_input_tokens()` now reads `prompt_tokens` alone. Verified on a live `databricks-glm-5-2` response where `prompt_tokens + completion == total` proves inclusivity. (Anthropic's native route genuinely *excludes* the cache fields and is still summed — the two never collide, because `claude*` models route to the Anthropic path.) ### 4. Proxy + TLS-trust passthrough into MCP tools (`mcp.rs`) — independent fix `buzz-agent` `env_clear()`s each MCP child, and the allowlist carried no proxy/TLS vars. On a proxy-only host that doesn't degrade the tools, it **blinds** them: apt, curl, pip, git connect directly, the egress firewall resets the socket, and the agent reports "Connection reset by peer" — indistinguishable from a genuinely offline task. Adds both spellings of `HTTP(S)_PROXY`/`NO_PROXY`/`ALL_PROXY` (curl/git read lowercase; Go/Python read uppercase; libcurl ignores uppercase `HTTP_PROXY`) plus `SSL_CERT_FILE`/`SSL_CERT_DIR` for TLS-terminating proxies that present their own CA. ## Testing - `cargo fmt --all -- --check`, `cargo clippy -p buzz-agent -p buzz-acp --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent -p buzz-acp` — **all green** (632 + 299 lib tests plus integration suites, 0 failures). New tests cover: the three breakpoints and the disabled/empty-system/single-message edge cases; nested-vs-flat cache parsing for all three routes; the Databricks inclusive-`prompt_tokens` fix; wire deserialization of `accumulatedCachedInputTokens`; and the proxy/TLS passthrough allowlist. - Pre-push lefthook suite green (branch-skew, rust-tests, test, desktop-check/test/tauri). ## Relationship to the benchmark branch These are the non-`benchmarks/` changes from `benchmark/harness-accounting-and-solo`, lifted onto a clean base off `main` so they can merge independently. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Atish Patel Co-authored-by: Claude Opus 4.8 (1M context) --- crates/buzz-acp/src/acp.rs | 5 + crates/buzz-acp/src/usage.rs | 40 ++++ crates/buzz-agent/src/agent.rs | 17 ++ crates/buzz-agent/src/config.rs | 9 + crates/buzz-agent/src/lib.rs | 24 +- crates/buzz-agent/src/llm.rs | 376 ++++++++++++++++++++++++++++++-- crates/buzz-agent/src/mcp.rs | 60 +++++ crates/buzz-agent/src/types.rs | 11 + 8 files changed, 519 insertions(+), 23 deletions(-) diff --git a/crates/buzz-acp/src/acp.rs b/crates/buzz-acp/src/acp.rs index 9eb668cbc2..3514580519 100644 --- a/crates/buzz-acp/src/acp.rs +++ b/crates/buzz-acp/src/acp.rs @@ -1848,6 +1848,11 @@ impl AcpClient { session_id = %notif.session_id, input = payload.accumulated_input_tokens, output = payload.accumulated_output_tokens, + // A subset of `input`, logged so downstream accounting can + // price it at the provider's cached rate. Always emitted, + // including as 0, so a parser can tell "no cache hits" + // apart from "this build predates the field". + cached = payload.accumulated_cached_input_tokens, "goose usage update" ); self.goose_usage.record(¬if.session_id, payload); diff --git a/crates/buzz-acp/src/usage.rs b/crates/buzz-acp/src/usage.rs index a4f7abd3b3..8cca9c96f8 100644 --- a/crates/buzz-acp/src/usage.rs +++ b/crates/buzz-acp/src/usage.rs @@ -85,6 +85,12 @@ pub(crate) struct UsageUpdatePayload { pub context_limit: u64, pub accumulated_input_tokens: u64, pub accumulated_output_tokens: u64, + /// The cache-served subset of `accumulated_input_tokens`. Optional — goose + /// does not send it, and buzz-agent only reports a non-zero value when the + /// provider returned a cache split, so `0` legitimately means either "no + /// cache hits" or "provider reported none". + #[serde(default)] + pub accumulated_cached_input_tokens: u64, pub accumulated_cost: Option, /// Effective model id for this turn. Optional — goose payloads that /// predate this field deserialize cleanly as `None`. @@ -323,12 +329,44 @@ impl UsageTracker { mod tests { use super::*; + /// The camelCase key buzz-agent actually puts on the wire must land on the + /// field. A rename mismatch here would deserialize to the serde default of + /// 0, and every trial would price as if nothing had ever been cached — the + /// exact silent failure this field was added to remove. + #[test] + fn cached_input_tokens_deserialize_from_the_wire_key() { + let p: UsageUpdatePayload = serde_json::from_value(serde_json::json!({ + "used": 15_247, + "contextLimit": 0, + "accumulatedInputTokens": 15_091, + "accumulatedOutputTokens": 156, + "accumulatedCachedInputTokens": 5_033, + })) + .expect("payload must deserialize"); + assert_eq!(p.accumulated_cached_input_tokens, 5_033); + assert!(p.accumulated_cached_input_tokens <= p.accumulated_input_tokens); + } + + /// goose does not send the field; its payloads must still deserialize. + #[test] + fn a_payload_without_the_cache_field_defaults_to_zero() { + let p: UsageUpdatePayload = serde_json::from_value(serde_json::json!({ + "used": 500, + "contextLimit": 200_000, + "accumulatedInputTokens": 400, + "accumulatedOutputTokens": 100, + })) + .expect("payload must deserialize without the cache field"); + assert_eq!(p.accumulated_cached_input_tokens, 0); + } + fn payload(input: u64, output: u64, cost: Option) -> UsageUpdatePayload { UsageUpdatePayload { used: input + output, context_limit: 200_000, accumulated_input_tokens: input, accumulated_output_tokens: output, + accumulated_cached_input_tokens: 0, accumulated_cost: cost, model: None, } @@ -340,6 +378,7 @@ mod tests { context_limit: 0, accumulated_input_tokens: input, accumulated_output_tokens: output, + accumulated_cached_input_tokens: 0, accumulated_cost: cost, model: None, } @@ -836,6 +875,7 @@ mod tests { context_limit: 200_000, accumulated_input_tokens: input, accumulated_output_tokens: output, + accumulated_cached_input_tokens: 0, accumulated_cost: cost, model: model.map(str::to_string), } diff --git a/crates/buzz-agent/src/agent.rs b/crates/buzz-agent/src/agent.rs index 730e87b2e8..cbf27357f7 100644 --- a/crates/buzz-agent/src/agent.rs +++ b/crates/buzz-agent/src/agent.rs @@ -60,6 +60,11 @@ pub struct RunCtx<'a> { /// Accumulated output tokens across all LLM rounds in this turn, for /// NIP-AM metric publishing. Reset to `None` at turn start in `run()`. pub turn_output_tokens: &'a mut Option, + /// The cache-served subset of `turn_input_tokens`, accumulated across all + /// LLM rounds in this turn. Reset to `None` at turn start in `run()`. + /// Consumers price this slice at the provider's cached rate; without it + /// every round of a growing conversation is billed at full price. + pub turn_cached_input_tokens: &'a mut Option, } impl RunCtx<'_> { @@ -78,6 +83,7 @@ impl RunCtx<'_> { // Reset per-turn token accumulators for this prompt. *self.turn_input_tokens = None; *self.turn_output_tokens = None; + *self.turn_cached_input_tokens = None; let mut round = 0u32; // Per-prompt `_Stop` objection count. Bounded per prompt (not per @@ -175,6 +181,17 @@ impl RunCtx<'_> { *self.turn_output_tokens = Some(self.turn_output_tokens.unwrap_or(0).saturating_add(out)); } + // Accumulate the cache-served subset of this turn's input. Tracked + // separately from `turn_input_tokens` rather than subtracted from + // it: the input total must stay inclusive for the handoff gate, + // which cares how much context was sent, not what it cost. + if let Some(cached) = response.cached_input_tokens { + *self.turn_cached_input_tokens = Some( + self.turn_cached_input_tokens + .unwrap_or(0) + .saturating_add(cached), + ); + } if !response.reasoning.is_empty() { wire::send( diff --git a/crates/buzz-agent/src/config.rs b/crates/buzz-agent/src/config.rs index 864fa7d237..037b67b3cb 100644 --- a/crates/buzz-agent/src/config.rs +++ b/crates/buzz-agent/src/config.rs @@ -738,6 +738,13 @@ pub struct Config { /// Thinking/reasoning effort level. `None` = use provider default (no /// thinking config sent). Set via `BUZZ_AGENT_THINKING_EFFORT`. pub thinking_effort: Option, + /// Emit Anthropic `cache_control` breakpoints on the stable prefix + /// (tools + system prompt) and the rolling conversation tail. Default on; + /// disable with `BUZZ_AGENT_PROMPT_CACHING=0`. Only consulted on Anthropic + /// Messages routes (first-party Anthropic and the DatabricksV2 Claude + /// route) — the Databricks gateway does not auto-cache, so without this the + /// surfaced `cache_read_input_tokens` is structurally always 0. + pub prompt_caching: bool, } impl Config { @@ -833,6 +840,7 @@ impl Config { hook_servers: parse_hook_servers_env("MCP_HOOK_SERVERS"), hints_enabled: parse_env("BUZZ_AGENT_NO_HINTS", 0u8)? == 0, thinking_effort: parse_thinking_effort(env("BUZZ_AGENT_THINKING_EFFORT").as_deref())?, + prompt_caching: parse_env("BUZZ_AGENT_PROMPT_CACHING", 1u8)? != 0, }; cfg.validate()?; Ok(cfg) @@ -874,6 +882,7 @@ impl Config { hook_servers: HookServers::None, hints_enabled: false, thinking_effort: None, + prompt_caching: false, } } diff --git a/crates/buzz-agent/src/lib.rs b/crates/buzz-agent/src/lib.rs index e141b9860f..6745dd0f92 100644 --- a/crates/buzz-agent/src/lib.rs +++ b/crates/buzz-agent/src/lib.rs @@ -100,6 +100,11 @@ struct Session { accumulated_input_tokens: u64, /// Session-cumulative output tokens across all turns. accumulated_output_tokens: u64, + /// Session-cumulative cache-served input tokens across all turns — a subset + /// of `accumulated_input_tokens`, not an addition to it. Emitted alongside + /// it so a consumer can price the cached slice at the provider's discounted + /// rate instead of assuming every input token cost full price. + accumulated_cached_input_tokens: u64, } fn die(msg: String) -> ! { @@ -426,6 +431,7 @@ async fn session_new(app: &Arc, id: Value, params: Value, wire_tx: &WireSen effective_model: None, accumulated_input_tokens: 0, accumulated_output_tokens: 0, + accumulated_cached_input_tokens: 0, }, ); drop(sessions); @@ -672,6 +678,7 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender .unwrap_or(&app.cfg.model); let mut turn_input_tokens: Option = None; let mut turn_output_tokens: Option = None; + let mut turn_cached_input_tokens: Option = None; let mut ctx = RunCtx { cfg: &app.cfg, effective_model: effective_model_str, @@ -690,6 +697,7 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender last_request_history_bytes: &mut last_request_history_bytes, turn_input_tokens: &mut turn_input_tokens, turn_output_tokens: &mut turn_output_tokens, + turn_cached_input_tokens: &mut turn_cached_input_tokens, }; let result = ctx.run(p.prompt).await; if let Some(s) = app.sessions.lock().await.get_mut(&sid) { @@ -722,14 +730,21 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender s.accumulated_output_tokens = s .accumulated_output_tokens .saturating_add(turn_output_tokens.unwrap_or(0)); - Some((s.accumulated_input_tokens, s.accumulated_output_tokens)) + s.accumulated_cached_input_tokens = s + .accumulated_cached_input_tokens + .saturating_add(turn_cached_input_tokens.unwrap_or(0)); + Some(( + s.accumulated_input_tokens, + s.accumulated_output_tokens, + s.accumulated_cached_input_tokens, + )) } else { // Session is gone — the accumulated baseline no longer exists, so // there is nothing correct to emit. Skip the usage notification. None } }; - if let Some((accumulated_in, accumulated_out)) = accumulated { + if let Some((accumulated_in, accumulated_out, accumulated_cached)) = accumulated { wire::send( &wire_tx, goose_session_update( @@ -742,6 +757,11 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender "contextLimit": 0u64, "accumulatedInputTokens": accumulated_in, "accumulatedOutputTokens": accumulated_out, + // A subset of accumulatedInputTokens, not an addition to + // it. Extends goose's usage_update shape; a consumer that + // does not know the field ignores it and prices exactly as + // it did before. + "accumulatedCachedInputTokens": accumulated_cached, "model": effective_model_str, }), ), diff --git a/crates/buzz-agent/src/llm.rs b/crates/buzz-agent/src/llm.rs index 23cef24e72..f2359f9ba3 100644 --- a/crates/buzz-agent/src/llm.rs +++ b/crates/buzz-agent/src/llm.rs @@ -720,6 +720,12 @@ fn anthropic_body( } } flush(&mut messages, &mut pending); + // Rolling cache breakpoint: mark the tail of the (append-only) conversation + // so the next turn re-reads this whole prefix from cache instead of paying + // full input price for it. See `stamp_rolling_cache_breakpoint`. + if cfg.prompt_caching { + stamp_rolling_cache_breakpoint(&mut messages); + } let tools_json: Vec = tools .iter() .map(|t| { @@ -727,8 +733,19 @@ fn anthropic_body( "name": t.name, "description": t.description, "input_schema": t.input_schema }) }) .collect(); + // Static prefix breakpoint: caching the `system` block caches the whole + // prefix up to and including it — and the prefix order is + // `tools -> system -> messages`, so this single marker caches tools + + // system together. Requires the structured (array) form of `system`; skip + // it for an empty prompt since Anthropic rejects empty text blocks. + let system_value = if cfg.prompt_caching && !system_prompt.is_empty() { + json!([{ "type": "text", "text": system_prompt, + "cache_control": { "type": "ephemeral" } }]) + } else { + json!(system_prompt) + }; let mut body = json!({ "model": effective_model, "max_tokens": cfg.max_output_tokens, - "system": system_prompt, "messages": messages }); + "system": system_value, "messages": messages }); if let Some(e) = effort { let (thinking, output_config) = crate::config::anthropic_thinking_config(effective_model, e, cfg.max_output_tokens); @@ -745,6 +762,41 @@ fn anthropic_body( body } +/// Attach ephemeral `cache_control` markers to the tail of the conversation so +/// the next turn re-reads the whole prior prefix from cache (~0.1x input price) +/// rather than re-billing it as fresh input. Anthropic caches the prefix up to +/// and including each marked block. +/// +/// We mark the last content block of the last *two* messages, not just the +/// final one. Each Anthropic breakpoint walks back at most 20 content blocks to +/// find a prior cache entry, and one agentic turn can append ~17 blocks at the +/// default `max_parallel_tools` (1 assistant text + N `tool_use` + +/// N `tool_result`). With only a tail marker, consecutive breakpoints sit a +/// full turn apart, which slips past the 20-block window as soon as parallelism +/// rises or a turn carries extra blocks — and the miss is silent. Marking the +/// last two messages halves the gap (to ~N+1 blocks), keeping a live cache +/// entry comfortably within reach. Uses 2 of the 4 allowed breakpoints; the +/// static `system` marker is the third. +/// +/// A no-op for messages whose content is empty or whose tail block is not a +/// JSON object. +fn stamp_rolling_cache_breakpoint(messages: &mut [Value]) { + let n = messages.len(); + // The two most-recently-appended messages (the current turn's tool results + // and the assistant turn before them). `checked_sub` + `flatten` skips the + // second index when there is only one message. + for idx in [n.checked_sub(1), n.checked_sub(2)].into_iter().flatten() { + if let Some(block) = messages[idx] + .get_mut("content") + .and_then(Value::as_array_mut) + .and_then(|c| c.last_mut()) + .and_then(Value::as_object_mut) + { + block.insert("cache_control".into(), json!({ "type": "ephemeral" })); + } + } +} + fn anthropic_tool_result_content(content: &[ToolResultContent]) -> Vec { content .iter() @@ -1079,11 +1131,18 @@ fn parse_responses(v: Value) -> Result { }; let input_tokens = sum_usage(&v, &["input_tokens"]); let output_tokens = sum_usage(&v, &["output_tokens"]); + // The Responses API nests the cache split under `input_tokens_details`. + let cached_input_tokens = usage_first( + &v, + &["cache_read_input_tokens"], + &[("input_tokens_details", "cached_tokens")], + ); Ok(LlmResponse { text, tool_calls, stop, input_tokens, + cached_input_tokens, output_tokens, reasoning, }) @@ -1131,19 +1190,70 @@ fn anthropic_input_tokens(v: &Value) -> Option { ) } -/// Input-token total for OpenAI Chat Completions and Databricks responses. -/// OpenAI's `prompt_tokens` is already inclusive. Databricks uses the same -/// `prompt_tokens` wire field but ALSO reports Anthropic-style cache fields -/// alongside it, so we sum them; the cache fields are simply absent (and -/// contribute 0) for vanilla OpenAI. +/// Input-token total for OpenAI Chat Completions and Databricks MLflow-route +/// responses. `prompt_tokens` is already the inclusive input total on both, so +/// it is read alone and never summed with the cache fields. +/// +/// Vanilla OpenAI nests the cache split under `prompt_tokens_details` and +/// `prompt_tokens` includes it. The Databricks MLflow route reports the split +/// with the flat Anthropic spelling (`cache_read_input_tokens`) *alongside* an +/// already-inclusive `prompt_tokens` — so summing double-counts. Verified on +/// `databricks-glm-5-2` (2026-07-28): `prompt_tokens 13320`, +/// `cache_read_input_tokens 13312`, `completion_tokens 30`, `total_tokens +/// 13350`; since `prompt_tokens + completion_tokens == total_tokens`, the 13312 +/// cached tokens are contained in the 13320, not additional to it. Summing gave +/// 26632 — nearly double — inflating both the context-budget gate and cost. +/// +/// This differs from Anthropic's native route (see [`anthropic_input_tokens`]), +/// where `input_tokens` genuinely EXCLUDES the cache fields and must be summed. +/// The two never collide here: the router sends `claude*` models to the +/// Anthropic route, so `parse_openai` only ever sees inclusive `prompt_tokens`. fn openai_chat_input_tokens(v: &Value) -> Option { - sum_usage( + sum_usage(v, &["prompt_tokens"]) +} + +/// First present value among `usage.` and `usage..` pairs. +/// +/// Cache counts are the one usage figure providers do not agree on the shape of. +/// Anthropic puts `cache_read_input_tokens` flat on `usage`; OpenAI nests the +/// same quantity one level down, under `prompt_tokens_details` on +/// `/chat/completions` and `input_tokens_details` on `/responses`. [`sum_usage`] +/// only reads flat keys, which is why the OpenAI split was invisible for so +/// long: `prompt_tokens` is already inclusive, so the *total* was right and +/// nothing looked broken while the discount silently went unclaimed. +/// +/// Returns the first candidate that resolves, not a sum — these are alternative +/// spellings of one number, so adding them would double-count on Databricks, +/// which reports both shapes. +fn usage_first(v: &Value, flat: &[&str], nested: &[(&str, &str)]) -> Option { + let usage = v.get("usage")?; + for f in flat { + if let Some(n) = usage.get(*f).and_then(Value::as_u64) { + return Some(n); + } + } + for (outer, leaf) in nested { + if let Some(n) = usage + .get(*outer) + .and_then(|o| o.get(*leaf)) + .and_then(Value::as_u64) + { + return Some(n); + } + } + None +} + +/// Cache-read tokens for an OpenAI Chat Completions response. +/// +/// `prompt_tokens_details.cached_tokens` is where vanilla OpenAI reports it. +/// The flat Anthropic spelling is checked first for Databricks, which routes +/// Anthropic models through an OpenAI-shaped envelope. +fn openai_chat_cached_tokens(v: &Value) -> Option { + usage_first( v, - &[ - "prompt_tokens", - "cache_read_input_tokens", - "cache_creation_input_tokens", - ], + &["cache_read_input_tokens"], + &[("prompt_tokens_details", "cached_tokens")], ) } @@ -1184,11 +1294,15 @@ fn parse_anthropic(v: Value) -> Result { } let input_tokens = anthropic_input_tokens(&v); let output_tokens = sum_usage(&v, &["output_tokens"]); + // Anthropic reports the cache split flat on `usage`. Note this is already + // part of `input_tokens` above, which sums it in deliberately. + let cached_input_tokens = usage_first(&v, &["cache_read_input_tokens"], &[]); Ok(LlmResponse { text, tool_calls, stop, input_tokens, + cached_input_tokens, output_tokens, reasoning, }) @@ -1235,11 +1349,13 @@ fn parse_openai(v: Value) -> Result { } let input_tokens = openai_chat_input_tokens(&v); let output_tokens = sum_usage(&v, &["completion_tokens"]); + let cached_input_tokens = openai_chat_cached_tokens(&v); Ok(LlmResponse { text, tool_calls, stop, input_tokens, + cached_input_tokens, output_tokens, reasoning, }) @@ -1606,6 +1722,7 @@ mod tests { prefer_mesh_for_auto: false, hints_enabled: true, thinking_effort: None, + prompt_caching: true, } } @@ -2604,6 +2721,100 @@ mod tests { assert_eq!(imgs[1]["image_url"]["url"], "data:image/png;base64,bbb"); } + // ---- prompt caching (cache_control) body-shape tests ---- + + #[test] + fn anthropic_body_stamps_cache_control_when_enabled() { + let body = anthropic_body( + &cfg(Provider::DatabricksV2), + "sys", + &[ + HistoryItem::User("hello".into()), + HistoryItem::Assistant { + text: "hi".into(), + tool_calls: vec![], + }, + HistoryItem::User("more".into()), + ], + &[], + "databricks-claude-opus-5", + None, + ); + // Static prefix: system promoted to a structured block carrying the marker. + assert_eq!(body["system"][0]["type"], "text"); + assert_eq!(body["system"][0]["text"], "sys"); + assert_eq!(body["system"][0]["cache_control"]["type"], "ephemeral"); + // Leapfrog: the last block of the last TWO messages is marked; earlier + // ones are not. Three distinct user/assistant/user turns → messages[1] + // and messages[2] marked, messages[0] clean. + let msgs = body["messages"].as_array().unwrap(); + assert_eq!(msgs.len(), 3); + let tail_block = |m: &Value| m["content"].as_array().unwrap().last().unwrap().clone(); + assert_eq!(tail_block(&msgs[2])["cache_control"]["type"], "ephemeral"); + assert_eq!(tail_block(&msgs[1])["cache_control"]["type"], "ephemeral"); + assert!( + tail_block(&msgs[0]).get("cache_control").is_none(), + "only the last two messages carry a breakpoint" + ); + } + + #[test] + fn anthropic_body_single_message_stamps_only_one_breakpoint() { + // With a single message there is no second turn to leapfrog to; the + // checked_sub(2) index is skipped rather than panicking. + let body = anthropic_body( + &cfg(Provider::DatabricksV2), + "sys", + &[HistoryItem::User("hello".into())], + &[], + "databricks-claude-opus-5", + None, + ); + let msgs = body["messages"].as_array().unwrap(); + assert_eq!(msgs.len(), 1); + assert_eq!( + msgs[0]["content"].as_array().unwrap().last().unwrap()["cache_control"]["type"], + "ephemeral" + ); + } + + #[test] + fn anthropic_body_no_cache_control_when_disabled() { + let mut c = cfg(Provider::DatabricksV2); + c.prompt_caching = false; + let body = anthropic_body( + &c, + "sys", + &[HistoryItem::User("hello".into())], + &[], + "databricks-claude-opus-5", + None, + ); + // system stays a bare string; no marker anywhere. + assert_eq!(body["system"], "sys"); + let last_block = &body["messages"][0]["content"][0]; + assert!(last_block.get("cache_control").is_none()); + } + + #[test] + fn anthropic_body_empty_system_stays_string_even_when_caching() { + // An empty system prompt must not become an empty text block — + // Anthropic rejects those. Caching is still applied to the tail. + let body = anthropic_body( + &cfg(Provider::DatabricksV2), + "", + &[HistoryItem::User("hello".into())], + &[], + "databricks-claude-opus-5", + None, + ); + assert_eq!(body["system"], ""); + assert_eq!( + body["messages"][0]["content"][0]["cache_control"]["type"], + "ephemeral" + ); + } + // ---- ThinkingEffort body-shape tests ---- #[test] @@ -3493,20 +3704,34 @@ mod tests { } #[test] - fn parse_openai_databricks_sums_cache_fields() { - // Databricks uses the OpenAI chat wire format (prompt_tokens) but also - // reports Anthropic-style cache fields; the inclusive total sums them. + fn parse_openai_databricks_prompt_tokens_already_inclusive() { + // Databricks' MLflow route uses the OpenAI chat wire format + // (prompt_tokens) but ALSO reports the flat Anthropic-style + // cache_read_input_tokens. prompt_tokens is already inclusive of that + // slice, so the total is prompt_tokens alone — summing double-counts. + // Values are the live databricks-glm-5-2 response (2026-07-28), where + // prompt_tokens + completion_tokens == total_tokens proves inclusivity. let v = serde_json::json!({ "choices": [{"finish_reason": "stop", "message": {"content": "hi"}}], "usage": { - "prompt_tokens": 200, - "completion_tokens": 4, - "total_tokens": 204, - "cache_read_input_tokens": 800, - "cache_creation_input_tokens": 0 + "prompt_tokens": 13320, + "completion_tokens": 30, + "total_tokens": 13350, + "cache_read_input_tokens": 13312, + "prompt_tokens_details": {"cached_tokens": 13312} } }); - assert_eq!(parse_openai(v).unwrap().input_tokens, Some(1000)); + let r = parse_openai(v).unwrap(); + assert_eq!( + r.input_tokens, + Some(13320), + "prompt_tokens is the inclusive total" + ); + assert_eq!(r.cached_input_tokens, Some(13312)); + assert!( + r.cached_input_tokens.unwrap() <= r.input_tokens.unwrap(), + "the cached slice is a subset of the input total" + ); } #[test] @@ -3517,6 +3742,115 @@ mod tests { assert_eq!(parse_openai(v).unwrap().input_tokens, None); } + #[test] + fn parse_openai_reads_nested_cached_tokens() { + // The shape vanilla OpenAI actually returns, captured from a live + // /chat/completions probe on gpt-5.6-luna: `prompt_tokens` is already + // inclusive and the cache split is nested one level down. Reading only + // flat keys left the discount unclaimed while the total looked correct, + // which is why this went unnoticed. + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "OK"}}], + "usage": { + "prompt_tokens": 5229, + "completion_tokens": 4, + "total_tokens": 5233, + "prompt_tokens_details": {"audio_tokens": 0, "cached_tokens": 5226, + "cache_write_tokens": 0} + } + }); + let r = parse_openai(v).unwrap(); + assert_eq!(r.input_tokens, Some(5229), "total must stay inclusive"); + assert_eq!(r.cached_input_tokens, Some(5226)); + } + + #[test] + fn parse_openai_cache_write_round_reports_zero_cached() { + // First request of a cold prefix: the provider writes the cache and + // serves nothing from it. `Some(0)` not `None` — the split was reported, + // it was simply zero, and a consumer must be able to tell the two apart. + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "OK"}}], + "usage": { + "prompt_tokens": 5229, + "completion_tokens": 4, + "prompt_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 5226} + } + }); + assert_eq!(parse_openai(v).unwrap().cached_input_tokens, Some(0)); + } + + #[test] + fn parse_openai_no_cache_detail_is_none() { + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "hi"}}], + "usage": {"prompt_tokens": 123, "completion_tokens": 4} + }); + assert_eq!(parse_openai(v).unwrap().cached_input_tokens, None); + } + + #[test] + fn parse_openai_prefers_flat_anthropic_spelling_over_nested() { + // Databricks reports both shapes for the same quantity. Take one, never + // the sum, or the cached slice double-counts. cache_read (800) is a + // subset of the inclusive prompt_tokens (1000), as it must be. + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "hi"}}], + "usage": { + "prompt_tokens": 1000, + "completion_tokens": 4, + "cache_read_input_tokens": 800, + "prompt_tokens_details": {"cached_tokens": 800} + } + }); + let r = parse_openai(v).unwrap(); + assert_eq!(r.input_tokens, Some(1000)); + assert_eq!(r.cached_input_tokens, Some(800)); + assert!(r.cached_input_tokens.unwrap() <= r.input_tokens.unwrap()); + } + + #[test] + fn parse_anthropic_reports_cache_read_as_cached() { + // Anthropic's `input_tokens` EXCLUDES cached, so the inclusive total is + // a sum -- but the cached slice must still be a subset of that total. + let v = serde_json::json!({ + "stop_reason": "end_turn", + "content": [{"type": "text", "text": "hi"}], + "usage": { + "input_tokens": 100, + "output_tokens": 7, + "cache_read_input_tokens": 900, + "cache_creation_input_tokens": 50 + } + }); + let r = parse_anthropic(v).unwrap(); + assert_eq!(r.input_tokens, Some(1050)); + assert_eq!(r.cached_input_tokens, Some(900)); + assert!(r.cached_input_tokens.unwrap() <= r.input_tokens.unwrap()); + } + + #[test] + fn parse_responses_reads_nested_cached_tokens() { + // The Responses API nests the same figure under a different key than + // /chat/completions does. + let v = serde_json::json!({ + "status": "completed", + "output": [{ + "type": "message", + "role": "assistant", + "content": [{"type": "output_text", "text": "hi"}] + }], + "usage": { + "input_tokens": 4000, + "output_tokens": 9, + "input_tokens_details": {"cached_tokens": 3584} + } + }); + let r = parse_responses(v).unwrap(); + assert_eq!(r.input_tokens, Some(4000)); + assert_eq!(r.cached_input_tokens, Some(3584)); + } + #[test] fn parse_responses_uses_input_tokens() { let v = serde_json::json!({ diff --git a/crates/buzz-agent/src/mcp.rs b/crates/buzz-agent/src/mcp.rs index 744b10dc7a..9ae125a0b7 100644 --- a/crates/buzz-agent/src/mcp.rs +++ b/crates/buzz-agent/src/mcp.rs @@ -52,6 +52,31 @@ const PASSTHROUGH_ENV: &[&str] = &[ "GIT_ASKPASS", "GIT_SSH_COMMAND", "GIT_CONFIG_GLOBAL", + // Proxy — on a host whose only route out is a CONNECT proxy, dropping + // these does not degrade the tools, it blinds them: apt, curl, pip and git + // all connect directly instead, and the egress firewall resets the socket. + // The agent then reports "Connection reset by peer" and concludes the + // environment has no network, which is indistinguishable in the transcript + // from a task that is genuinely offline. + // + // Both cases are needed. curl and git read the lowercase spellings, most + // Go and Python tooling reads the uppercase ones, and libcurl deliberately + // ignores uppercase HTTP_PROXY (CGI ambiguity), so keeping only one form + // silently breaks half the toolchain. + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "ALL_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "all_proxy", + // TLS trust — a proxy that terminates TLS presents its own CA, and an + // image whose trust store does not carry it fails every https fetch with a + // verification error. Same class of failure as the proxy vars: the parent + // was configured correctly and the child could not see it. + "SSL_CERT_FILE", + "SSL_CERT_DIR", // Buzz identity — dev-mcp writes NOSTR_PRIVATE_KEY to a keyfile then // removes it from its own env (children never see it). BUZZ_PRIVATE_KEY // and BUZZ_RELAY_URL are kept for the buzz CLI. BUZZ_AUTH_TAG is a @@ -1015,6 +1040,41 @@ mod content_tests { fn passthrough_includes_buzz_owner_attestation() { assert!(PASSTHROUGH_ENV.contains(&"BUZZ_AUTH_TAG")); } + + #[test] + fn passthrough_carries_proxy_configuration_to_tools() { + // On a proxy-only host this is the difference between an agent that can + // install a package and one that reports the network is down. Both + // spellings: libcurl ignores uppercase HTTP_PROXY, and Go/Python + // tooling largely ignores the lowercase set. + for var in [ + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "ALL_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "all_proxy", + ] { + assert!( + PASSTHROUGH_ENV.contains(&var), + "{var} must survive env_clear() or every MCP tool loses the proxy" + ); + } + } + + #[test] + fn passthrough_carries_tls_trust_to_tools() { + // A TLS-terminating proxy presents its own CA; without these the child + // rejects every https fetch even though the proxy itself is reachable. + for var in ["SSL_CERT_FILE", "SSL_CERT_DIR"] { + assert!( + PASSTHROUGH_ENV.contains(&var), + "{var} must survive env_clear() or https fails inside tools" + ); + } + } use rmcp::model::Content; #[cfg(windows)] diff --git a/crates/buzz-agent/src/types.rs b/crates/buzz-agent/src/types.rs index d29e975e03..1b5f30b1ce 100644 --- a/crates/buzz-agent/src/types.rs +++ b/crates/buzz-agent/src/types.rs @@ -139,6 +139,17 @@ pub struct LlmResponse { /// tokens, so reading it alone would undercount). Used to gate handoff on /// the real token budget rather than a byte estimate. pub input_tokens: Option, + /// The portion of `input_tokens` the provider served from its prompt cache, + /// or `None` when the response reported no cache split. Providers bill this + /// slice at a large discount (roughly 10x for both OpenAI and Anthropic), + /// so a consumer that prices all of `input_tokens` at the full rate + /// *overstates* cost — by a lot on an append-only agent loop, where most of + /// each request is a prefix the provider already has. + /// + /// This is a subset of `input_tokens`, never an addition to it: every + /// provider we speak to reports an inclusive input total, so adding this + /// would double-count. + pub cached_input_tokens: Option, /// Output tokens the provider reported for this request, or `None` if the /// response carried no usage. Used to accumulate per-turn output counts /// for NIP-AM metric publishing. From ce01e930edff97bdc12a205fb8f938fcacdba8c1 Mon Sep 17 00:00:00 2001 From: klopez4212 Date: Wed, 29 Jul 2026 15:35:41 +0100 Subject: [PATCH 02/32] Polish mobile typing indicator (#3528) ## Summary - Present channel and thread typing status in a composer-matched container. - Animate the strip so the message list moves smoothly as typing begins and ends. - Increase typing-label contrast and avatar/padding for readability. ## Pixel 10 snapshot ![Typing indicator above the composer](https://raw.githubusercontent.com/block/buzz/31de9f86a76fe61498bc7f2931d9e574827a9aa2/pr-3528--typing-indicator.png) ## Validation - `flutter test test/features/channels/channel_detail_page_test.dart` - `flutter analyze` Signed-off-by: kenny lopez --- .../channels/channel_detail_page.dart | 14 ++- .../channels/channel_detail_page/app_bar.dart | 64 ------------- .../channels/channel_typing_indicator.dart | 89 +++++++++++++++++++ .../features/channels/thread_detail_page.dart | 77 +++------------- .../channels/channel_detail_page_test.dart | 19 ++++ 5 files changed, 131 insertions(+), 132 deletions(-) create mode 100644 mobile/lib/features/channels/channel_typing_indicator.dart diff --git a/mobile/lib/features/channels/channel_detail_page.dart b/mobile/lib/features/channels/channel_detail_page.dart index 6bbb60c0d1..4304705f90 100644 --- a/mobile/lib/features/channels/channel_detail_page.dart +++ b/mobile/lib/features/channels/channel_detail_page.dart @@ -27,6 +27,7 @@ import 'agent_activity/working_bots_provider.dart'; import 'channel_management_provider.dart'; import 'channel_messages_provider.dart'; import 'channel_typing_provider.dart'; +import 'channel_typing_indicator.dart'; import 'channels_provider.dart'; import 'compose_bar.dart'; import 'date_formatters.dart'; @@ -368,8 +369,17 @@ class ChannelDetailPage extends HookConsumerWidget { ), ), ), - if (!resolvedChannel.isForum && typingEntries.isNotEmpty) - _TypingIndicator(entries: typingEntries), + if (!resolvedChannel.isForum) + AnimatedSize( + duration: MediaQuery.disableAnimationsOf(context) + ? Duration.zero + : const Duration(milliseconds: 180), + curve: Curves.easeOutCubic, + alignment: Alignment.bottomCenter, + child: typingEntries.isEmpty + ? const SizedBox.shrink() + : ChannelTypingIndicator(entries: typingEntries), + ), if (!resolvedChannel.isForum && resolvedChannel.isMember && !resolvedChannel.isArchived) diff --git a/mobile/lib/features/channels/channel_detail_page/app_bar.dart b/mobile/lib/features/channels/channel_detail_page/app_bar.dart index fbf5abfb3b..406d68b4f5 100644 --- a/mobile/lib/features/channels/channel_detail_page/app_bar.dart +++ b/mobile/lib/features/channels/channel_detail_page/app_bar.dart @@ -19,70 +19,6 @@ double _dmAppBarTitleContentHeight(BuildContext context) { return textHeight > 30 ? textHeight : 30; } -class _TypingIndicator extends ConsumerWidget { - final List entries; - - const _TypingIndicator({required this.entries}); - - @override - Widget build(BuildContext context, WidgetRef ref) { - final userCache = ref.watch(userCacheProvider); - final names = entries.map((e) { - final profile = - userCache[e.pubkey.toLowerCase()] ?? - ref.read(userCacheProvider.notifier).get(e.pubkey.toLowerCase()); - return profile?.label ?? shortPubkey(e.pubkey); - }).toList(); - final text = switch (names.length) { - 1 => '${names[0]} is typing…', - 2 => '${names[0]} and ${names[1]} are typing…', - _ => '${names[0]} and ${names.length - 1} others are typing…', - }; - - final visibleEntries = entries.take(3).toList(); - final avatarCount = visibleEntries.length; - - return Container( - width: double.infinity, - padding: const EdgeInsets.symmetric( - horizontal: Grid.gutter, - vertical: Grid.quarter + 2, - ), - child: Row( - children: [ - SizedBox( - width: 20.0 + (avatarCount - 1) * 12.0, - height: 20, - child: Stack( - children: [ - for (var i = 0; i < avatarCount; i++) - Positioned( - left: i * 12.0, - child: SmallAvatar( - pubkey: visibleEntries[i].pubkey, - userCache: userCache, - ), - ), - ], - ), - ), - const SizedBox(width: Grid.xxs), - Flexible( - child: Text( - text, - style: context.textTheme.labelSmall?.copyWith( - color: context.colors.outline, - fontStyle: FontStyle.italic, - ), - overflow: TextOverflow.ellipsis, - ), - ), - ], - ), - ); - } -} - class _MembersButton extends ConsumerWidget { final String channelId; final Channel channel; diff --git a/mobile/lib/features/channels/channel_typing_indicator.dart b/mobile/lib/features/channels/channel_typing_indicator.dart new file mode 100644 index 0000000000..d021b9e287 --- /dev/null +++ b/mobile/lib/features/channels/channel_typing_indicator.dart @@ -0,0 +1,89 @@ +import 'package:flutter/material.dart'; +import 'package:hooks_riverpod/hooks_riverpod.dart'; + +import '../../shared/theme/theme.dart'; +import '../../shared/utils/string_utils.dart'; +import '../profile/user_cache_provider.dart'; +import 'channel_typing_provider.dart'; +import 'small_avatar.dart'; + +/// Composer-adjacent status for people currently typing in a channel or thread. +class ChannelTypingIndicator extends ConsumerWidget { + final List entries; + + const ChannelTypingIndicator({super.key, required this.entries}); + + @override + Widget build(BuildContext context, WidgetRef ref) { + final userCache = ref.watch(userCacheProvider); + final names = entries.map((entry) { + final profile = + userCache[entry.pubkey.toLowerCase()] ?? + ref.read(userCacheProvider.notifier).get(entry.pubkey.toLowerCase()); + return profile?.label ?? shortPubkey(entry.pubkey); + }).toList(); + final text = switch (names.length) { + 1 => '${names[0]} is typing…', + 2 => '${names[0]} and ${names[1]} are typing…', + _ => '${names[0]} and ${names.length - 1} others are typing…', + }; + final visibleEntries = entries.take(3).toList(); + final avatarCount = visibleEntries.length; + + return Padding( + padding: const EdgeInsets.only( + left: Grid.twelve, + right: Grid.twelve, + bottom: Grid.xxs, + ), + child: Container( + key: const ValueKey('channel-typing-indicator'), + width: double.infinity, + padding: const EdgeInsets.symmetric( + horizontal: Grid.xxs, + vertical: Grid.xxs, + ), + decoration: BoxDecoration( + color: context.colors.surfaceContainerHighest, + borderRadius: BorderRadius.circular(Radii.dialog), + border: Border.all( + color: Colors.black.withValues(alpha: 0.04), + width: 1, + ), + ), + child: Row( + children: [ + SizedBox( + width: 24.0 + (avatarCount - 1) * 14.0, + height: 24, + child: Stack( + children: [ + for (var i = 0; i < avatarCount; i++) + Positioned( + left: i * 14.0, + child: SmallAvatar( + pubkey: visibleEntries[i].pubkey, + userCache: userCache, + size: 24, + ), + ), + ], + ), + ), + const SizedBox(width: Grid.xxs), + Flexible( + child: Text( + text, + style: context.textTheme.labelSmall?.copyWith( + color: context.colors.primary, + fontStyle: FontStyle.italic, + ), + overflow: TextOverflow.ellipsis, + ), + ), + ], + ), + ), + ); + } +} diff --git a/mobile/lib/features/channels/thread_detail_page.dart b/mobile/lib/features/channels/thread_detail_page.dart index 0babba2394..167b111f4f 100644 --- a/mobile/lib/features/channels/thread_detail_page.dart +++ b/mobile/lib/features/channels/thread_detail_page.dart @@ -12,6 +12,7 @@ import '../profile/user_cache_provider.dart'; import '../profile/user_profile.dart'; import 'channel_link_navigation.dart'; import 'channel_typing_provider.dart'; +import 'channel_typing_indicator.dart'; import 'thread_replies_provider.dart'; import 'channels_provider.dart'; import 'compose_bar.dart'; @@ -280,8 +281,16 @@ class ThreadDetailPage extends HookConsumerWidget { }, ), ), - if (threadTyping.isNotEmpty) - _ThreadTypingIndicator(entries: threadTyping), + AnimatedSize( + duration: MediaQuery.disableAnimationsOf(context) + ? Duration.zero + : const Duration(milliseconds: 180), + curve: Curves.easeOutCubic, + alignment: Alignment.bottomCenter, + child: threadTyping.isEmpty + ? const SizedBox.shrink() + : ChannelTypingIndicator(entries: threadTyping), + ), if (isMember && !isArchived) ComposeBar( channelId: channelId, @@ -663,70 +672,6 @@ class _ThreadMessage extends ConsumerWidget { } } -class _ThreadTypingIndicator extends ConsumerWidget { - final List entries; - - const _ThreadTypingIndicator({required this.entries}); - - @override - Widget build(BuildContext context, WidgetRef ref) { - final userCache = ref.watch(userCacheProvider); - final names = entries.map((e) { - final profile = - userCache[e.pubkey.toLowerCase()] ?? - ref.read(userCacheProvider.notifier).get(e.pubkey.toLowerCase()); - return profile?.label ?? shortPubkey(e.pubkey); - }).toList(); - final text = switch (names.length) { - 1 => '${names[0]} is typing...', - 2 => '${names[0]} and ${names[1]} are typing...', - _ => '${names[0]} and ${names.length - 1} others are typing...', - }; - - final visibleEntries = entries.take(3).toList(); - final avatarCount = visibleEntries.length; - - return Container( - width: double.infinity, - padding: const EdgeInsets.symmetric( - horizontal: Grid.gutter, - vertical: Grid.quarter + 2, - ), - child: Row( - children: [ - SizedBox( - width: 20.0 + (avatarCount - 1) * 12.0, - height: 20, - child: Stack( - children: [ - for (var i = 0; i < avatarCount; i++) - Positioned( - left: i * 12.0, - child: SmallAvatar( - pubkey: visibleEntries[i].pubkey, - userCache: userCache, - ), - ), - ], - ), - ), - const SizedBox(width: Grid.xxs), - Flexible( - child: Text( - text, - style: context.textTheme.labelSmall?.copyWith( - color: context.colors.outline, - fontStyle: FontStyle.italic, - ), - overflow: TextOverflow.ellipsis, - ), - ), - ], - ), - ); - } -} - class _Avatar extends StatelessWidget { final UserProfile? profile; final String pubkey; diff --git a/mobile/test/features/channels/channel_detail_page_test.dart b/mobile/test/features/channels/channel_detail_page_test.dart index 48ad861d3a..127e6851e6 100644 --- a/mobile/test/features/channels/channel_detail_page_test.dart +++ b/mobile/test/features/channels/channel_detail_page_test.dart @@ -1808,6 +1808,25 @@ void main() { await tester.pumpAndSettle(); expect(find.text('Alice is typing…'), findsOneWidget); + + final indicator = tester.widget( + find.byKey(const ValueKey('channel-typing-indicator')), + ); + final decoration = indicator.decoration! as BoxDecoration; + expect( + indicator.padding, + const EdgeInsets.symmetric(horizontal: Grid.xxs, vertical: Grid.xxs), + ); + expect( + decoration.color, + AppTheme.light().colorScheme.surfaceContainerHighest, + ); + expect(decoration.border, isA()); + expect( + tester.widget(find.text('Alice is typing…')).style?.color, + AppTheme.light().colorScheme.primary, + ); + expect(tester.widget(find.byType(SmallAvatar)).size, 24); }); testWidgets('shows two typers', (tester) async { From 24d90d1280a9325c6cbcf8eea30ac54db5afd2cb Mon Sep 17 00:00:00 2001 From: klopez4212 Date: Wed, 29 Jul 2026 15:42:17 +0100 Subject: [PATCH 03/32] Refine community invite limits (#3529) ## Summary - Simplify the community invite dialog around link sharing. - Add matching expiry and use-limit dropdowns, with sensible preset use caps. - Cover the default unlimited and selected-limit invite payloads. ## Validation - `pnpm -C desktop run build:e2e` - `pnpm -C desktop exec playwright test tests/e2e/invite-link-copy.spec.ts tests/e2e/invites-settings-screenshots.spec.ts --project=smoke` Signed-off-by: kenny lopez --- .../ui/CommunityInviteDialog.tsx | 7 +- .../ui/InviteLinkSection.tsx | 181 +++++++++--------- desktop/tests/e2e/invite-link-copy.spec.ts | 30 +++ .../e2e/invites-settings-screenshots.spec.ts | 12 +- 4 files changed, 134 insertions(+), 96 deletions(-) diff --git a/desktop/src/features/community-members/ui/CommunityInviteDialog.tsx b/desktop/src/features/community-members/ui/CommunityInviteDialog.tsx index bd23e2bbec..9daca590f4 100644 --- a/desktop/src/features/community-members/ui/CommunityInviteDialog.tsx +++ b/desktop/src/features/community-members/ui/CommunityInviteDialog.tsx @@ -3,6 +3,7 @@ import * as React from "react"; import { Dialog, DialogContent, + DialogDescription, DialogHeader, DialogTitle, } from "@/shared/ui/dialog"; @@ -29,12 +30,14 @@ export function CommunityInviteDialog({ return ( - + Invite to community + + Anyone with this link can join this community. + diff --git a/desktop/src/features/community-members/ui/InviteLinkSection.tsx b/desktop/src/features/community-members/ui/InviteLinkSection.tsx index c4e140f723..dc0735c85e 100644 --- a/desktop/src/features/community-members/ui/InviteLinkSection.tsx +++ b/desktop/src/features/community-members/ui/InviteLinkSection.tsx @@ -8,16 +8,12 @@ import { Button } from "@/shared/ui/button"; import { DropdownMenu, DropdownMenuContent, - DropdownMenuLabel, DropdownMenuRadioGroup, DropdownMenuRadioItem, - DropdownMenuSeparator, DropdownMenuTrigger, } from "@/shared/ui/dropdown-menu"; -import { Input } from "@/shared/ui/input"; import { Separator } from "@/shared/ui/separator"; import { Spinner } from "@/shared/ui/spinner"; -import { Switch } from "@/shared/ui/switch"; const TTL_OPTIONS: { label: string; value: number }[] = [ { label: "1 day", value: 24 * 60 * 60 }, @@ -26,6 +22,15 @@ const TTL_OPTIONS: { label: string; value: number }[] = [ { label: "30 days", value: 30 * 24 * 60 * 60 }, ]; +const MAX_USE_OPTIONS: { label: string; value: number | null }[] = [ + { label: "No limit", value: null }, + { label: "1 use", value: 1 }, + { label: "3 uses", value: 3 }, + { label: "5 uses", value: 5 }, + { label: "10 uses", value: 10 }, + { label: "25 uses", value: 25 }, +]; + export const DEFAULT_INVITE_TTL_SECS = TTL_OPTIONS[1].value; type CopyStatus = "idle" | "copying" | "copied"; @@ -45,16 +50,12 @@ export function InviteLinkSection({ ttlSecs: number; }) { const [copyStatus, setCopyStatus] = React.useState("idle"); - const [maxUsesEnabled, setMaxUsesEnabled] = React.useState(true); - const [maxUsesInput, setMaxUsesInput] = React.useState("3"); - const parsedMaxUses = Number(maxUsesInput); - const maxUsesValid = - !maxUsesEnabled || - (Number.isInteger(parsedMaxUses) && - parsedMaxUses >= 1 && - parsedMaxUses <= 10000); + const [maxUses, setMaxUses] = React.useState(null); const ttlLabel = TTL_OPTIONS.find((option) => option.value === ttlSecs)?.label ?? "3 days"; + const maxUsesLabel = + MAX_USE_OPTIONS.find((option) => option.value === maxUses)?.label ?? + "No limit"; const copyLabel = copyStatus === "copying" ? "Copying…" @@ -69,13 +70,10 @@ export function InviteLinkSection({ }, [copyStatus]); async function handleCopy() { - if (copyStatus === "copying" || !maxUsesValid) return; + if (copyStatus === "copying") return; setCopyStatus("copying"); try { - const invite = await mintInvite({ - ttlSecs, - maxUses: maxUsesEnabled ? parsedMaxUses : null, - }); + const invite = await mintInvite({ ttlSecs, maxUses }); await writeTextToClipboard(invite.url); setCopyStatus("copied"); toast.success("Invite link copied"); @@ -87,82 +85,79 @@ export function InviteLinkSection({ return (
-
- - -
-

Share with a link

-

- Anyone with the link can join this community. -

+
+
+ Expires after + + + + + + onTtlSecsChange(Number(value))} + value={String(ttlSecs)} + > + {TTL_OPTIONS.map((option) => ( + + {option.label} + + ))} + + + +
+
+ Limit number of uses + + + + + + + setMaxUses(value === "no-limit" ? null : Number(value)) + } + value={String(maxUses ?? "no-limit")} + > + {MAX_USE_OPTIONS.map((option) => ( + + {option.label} + + ))} + + +
- - - - - - Expires after - - onTtlSecsChange(Number(value))} - value={String(ttlSecs)} - > - {TTL_OPTIONS.map((option) => ( - - {option.label} - - ))} - - - -
-
- - - {maxUsesEnabled ? ( - setMaxUsesInput(event.target.value)} - placeholder="3" - type="number" - value={maxUsesInput} - /> - ) : null} - {maxUsesEnabled && !maxUsesValid ? ( - - Enter a whole number from 1 to 10,000 - - ) : null}
@@ -170,7 +165,7 @@ export function InviteLinkSection({ className="shrink-0 border-border shadow-none" data-copy-status={copyStatus} data-testid="copy-invite-link" - disabled={copyStatus === "copying" || !maxUsesValid} + disabled={copyStatus === "copying"} onClick={() => void handleCopy()} size="sm" type="button" diff --git a/desktop/tests/e2e/invite-link-copy.spec.ts b/desktop/tests/e2e/invite-link-copy.spec.ts index 94bc8e26b0..015abd848f 100644 --- a/desktop/tests/e2e/invite-link-copy.spec.ts +++ b/desktop/tests/e2e/invite-link-copy.spec.ts @@ -3,7 +3,10 @@ import { expect, test } from "@playwright/test"; import { installMockBridge } from "../helpers/bridge"; import { openSettings } from "../helpers/settings"; +let invitePayloads: Record[]; + test.beforeEach(async ({ page }) => { + invitePayloads = []; await page.context().grantPermissions(["clipboard-read", "clipboard-write"], { origin: "http://127.0.0.1:4173", }); @@ -11,6 +14,7 @@ test.beforeEach(async ({ page }) => { relayRequiresMembership: true, }); await page.route("**/api/invites", async (route) => { + invitePayloads.push(route.request().postDataJSON()); await route.fulfill({ contentType: "application/json", json: { @@ -33,8 +37,12 @@ test("copies a freshly minted invite link without showing a URL or QR code", asy await page.getByTestId("community-invite-dialog-trigger").click(); await expect(page.getByTestId("invite-link-url")).toHaveCount(0); await expect(page.getByTestId("invite-link-qr-code")).toHaveCount(0); + await expect(page.getByTestId("invite-link-max-uses-trigger")).toHaveText( + "No limit", + ); await page.getByTestId("copy-invite-link").click(); await expect(page.getByTestId("copy-invite-link")).toContainText("Copied"); + expect(invitePayloads).toEqual([{ ttl_secs: 3 * 24 * 60 * 60 }]); const payload = await page.evaluate(() => { const log = ( @@ -53,3 +61,25 @@ test("copies a freshly minted invite link without showing a URL or QR code", asy text: "buzz://join?relay=wss%3A%2F%2Frelay.example.com&code=qr-download-test", }); }); + +test("sets a selected invite-use limit", async ({ page }) => { + await page.goto("/"); + await openSettings(page, "community-members"); + await page.getByTestId("community-invite-dialog-trigger").click(); + + const maxUsesTrigger = page.getByTestId("invite-link-max-uses-trigger"); + await maxUsesTrigger.click(); + await expect( + page.getByRole("menuitemradio", { name: "No limit" }), + ).toBeVisible(); + await expect( + page.getByRole("menuitemradio", { name: "25 uses" }), + ).toBeVisible(); + await page.getByTestId("invite-link-max-uses-10").click(); + await expect(maxUsesTrigger).toHaveText("10 uses"); + await page.getByTestId("copy-invite-link").click(); + await expect(page.getByTestId("copy-invite-link")).toContainText("Copied"); + expect(invitePayloads).toEqual([ + { max_uses: 10, ttl_secs: 3 * 24 * 60 * 60 }, + ]); +}); diff --git a/desktop/tests/e2e/invites-settings-screenshots.spec.ts b/desktop/tests/e2e/invites-settings-screenshots.spec.ts index 654aec56fe..c56e33bcd5 100644 --- a/desktop/tests/e2e/invites-settings-screenshots.spec.ts +++ b/desktop/tests/e2e/invites-settings-screenshots.spec.ts @@ -78,15 +78,25 @@ test("capture: share-style community invite dialog", async ({ page }) => { await expect(page.getByTestId("community-invite-email-field")).toHaveCount(0); await expect(page.getByPlaceholder("Type an email address")).toHaveCount(0); await expect( - dialog.getByRole("heading", { name: "Share with a link" }), + dialog.getByText("Anyone with this link can join this community."), ).toBeVisible(); + await expect(dialog.getByText("Expires after")).toBeVisible(); + await expect(dialog.getByText("Limit number of uses")).toBeVisible(); + await expect(page.getByTestId("invite-link-max-uses-trigger")).toHaveText( + "No limit", + ); await expect(page.getByTestId("copy-invite-link")).toHaveText("Copy link"); await expect(page.getByTestId("invite-link-qr-code")).toHaveCount(0); await expect(page.getByTestId("invite-link-url")).toHaveCount(0); const expiryTrigger = page.getByTestId("invite-link-ttl-trigger"); await expect(expiryTrigger).toHaveText("3 days"); + await expect(expiryTrigger).toHaveCSS("font-size", "14px"); + await expect( + dialog.getByText("Limit number of uses", { exact: true }), + ).toHaveCSS("font-size", "14px"); await expiryTrigger.click(); + await expect(page.getByRole("menu")).not.toContainText("Expires after"); await expect( page.getByRole("menuitemradio", { name: "1 day" }), ).toBeVisible(); From 6438dedf83a9dbe1853e484326911bf6c7f1618c Mon Sep 17 00:00:00 2001 From: Atish Patel Date: Wed, 29 Jul 2026 10:02:12 -0500 Subject: [PATCH 04/32] feat(agent): route Claude/GPT model families to their native gateway wire (#3538) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Databricks v2 chooses the gateway wire format — OpenAI Responses, Anthropic Messages, or MLflow chat — purely from substrings in the endpoint name. There is no family field on the endpoint to key off, so the substring set *is* the routing contract. The matcher only recognised `gpt-5`/`gpt5` and `claude`, which makes correct billing depend on every Claude endpoint happening to be named with the literal string "claude". ## Why this matters Getting a Claude model onto the Anthropic Messages route is exactly what lets buzz attach the `cache_control` breakpoint (the fix in #3463). If a Claude endpoint's catalog name omits "claude" — an alias, a bare `opus-5`, a `goose-opus-5` — it silently falls through to the MLflow (OpenAI-wire) path, where Anthropic prompt caching is **structurally impossible**. The result is the same failure #3463 fixed: 0% cache reads, the full ~10x read discount lost, and no error — a naming convention quietly holding up a billing-correctness invariant. ## What changed `databricks_v2_route_for_model` (`crates/buzz-agent/src/llm.rs`) now matches broader, case-insensitive marker sets: - **Claude → Anthropic Messages:** `claude`, `opus`, `sonnet`, `haiku`, `mythos`, `fable` — the Claude family names and release code names, so a Claude endpoint reaches the cache-capable route regardless of how it's named. - **GPT → OpenAI Responses:** the `gpt` family (now `gpt` on its own, not just `gpt-5`) plus the GPT-5 launch code names `sol`, `luna`, `terra`. OpenAI markers are evaluated first, preserving the prior `gpt-5`-first precedence for any name that could carry both. Names matching neither set still fall through to the MLflow chat route. ## Testing - `cargo fmt`, `cargo clippy -p buzz-agent --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent` — all green (299 lib + integration suites, 0 failures). The `databricks_v2_routes_by_model_family` test was expanded to cover each new marker, the GPT-5 code names, case-insensitivity, and the unchanged MLflow fallback (including `gemini`). ## Relationship to #3463 #3463 taught the Anthropic path to request caching; this makes sure Claude models actually land on that path. Follow-up still open: surfacing `cache_creation_input_tokens` end-to-end so a persistent `reads == 0 && writes == 0` reveals a disabled cache regardless of which wire a model takes — happy to do that next. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Atish Patel Co-authored-by: Claude Opus 4.8 (1M context) --- crates/buzz-agent/src/llm.rs | 141 +++++++++++++++++++++++++++++++++-- 1 file changed, 133 insertions(+), 8 deletions(-) diff --git a/crates/buzz-agent/src/llm.rs b/crates/buzz-agent/src/llm.rs index f2359f9ba3..f69a963533 100644 --- a/crates/buzz-agent/src/llm.rs +++ b/crates/buzz-agent/src/llm.rs @@ -1019,13 +1019,53 @@ fn is_responses_required_error(body: &str) -> bool { || b.contains("use the responses api") } +/// OpenAI-family code names that appear as their own segment in a Databricks v2 +/// endpoint name (the GPT-5 launch aliases). The `gpt` family itself is matched +/// separately by segment prefix so `gpt`, `gpt5`, and the `gpt` of a split +/// `gpt-5` all qualify. +const DATABRICKS_V2_OPENAI_CODE_NAMES: &[&str] = &["sol", "luna", "terra"]; + +/// Anthropic (Claude) family and release code names that appear as their own +/// segment in a Databricks v2 endpoint name — the `claude` prefix, the family +/// names (`opus`, `sonnet`, `haiku`), and the release code names (`mythos`, +/// `fable`). Getting a Claude model onto the Anthropic Messages route is what +/// lets it carry a `cache_control` breakpoint; an endpoint that matches none of +/// these falls through to the MLflow (OpenAI-wire) path, where Anthropic prompt +/// caching is structurally impossible and the discount is silently lost. +const DATABRICKS_V2_CLAUDE_NAMES: &[&str] = + &["claude", "opus", "sonnet", "haiku", "mythos", "fable"]; + +/// Split a Databricks v2 endpoint name into its lowercase alphanumeric segments, +/// breaking on any non-alphanumeric delimiter (`-`, `_`, `.`, `/`, …). E.g. +/// `Databricks-Claude-Opus-5` -> `["databricks", "claude", "opus", "5"]`. +fn model_name_segments(model: &str) -> Vec { + model + .split(|c: char| !c.is_ascii_alphanumeric()) + .filter(|s| !s.is_empty()) + .map(str::to_ascii_lowercase) + .collect() +} + fn databricks_v2_route_for_model(model: &str) -> DatabricksV2Route { - // Databricks v2 catalog names currently identify OpenAI-shaped GPT-5 - // models and Anthropic-shaped Claude models by these substrings. - let lower = model.to_ascii_lowercase(); - if lower.contains("gpt-5") || lower.contains("gpt5") { + // The v2 catalog exposes no family field, so the wire format is inferred + // from the endpoint name. Discovery deliberately keeps arbitrary custom + // aliases, so we match whole name *segments* rather than raw substrings: a + // substring test would misroute unrelated names — `consolidated-llama` + // (`sol`), `terraform-coder` (`terra`), `corpus-reranker`/`octopus-model` + // (`opus`) — onto a wire whose request shape their backend can't parse, + // turning a caching optimization into a hard request/parse failure. Segment + // matching still accepts real prefixed names like `goose-opus-5`. + let segments = model_name_segments(model); + let has_named_segment = + |names: &[&str]| segments.iter().any(|seg| names.contains(&seg.as_str())); + // `gpt` family: any segment beginning with `gpt` — covers `gpt`, `gpt5`, and + // the `gpt` segment of a split `gpt-5`, without matching mid-word. + let is_gpt_family = segments.iter().any(|seg| seg.starts_with("gpt")); + // OpenAI is checked before Claude so a name carrying both markers resolves + // to the OpenAI wire (preserving the prior `gpt-5`-first precedence). + if is_gpt_family || has_named_segment(DATABRICKS_V2_OPENAI_CODE_NAMES) { DatabricksV2Route::OpenAiResponses - } else if lower.contains("claude") { + } else if has_named_segment(DATABRICKS_V2_CLAUDE_NAMES) { DatabricksV2Route::AnthropicMessages } else { DatabricksV2Route::MlflowChatCompletions @@ -2579,20 +2619,105 @@ mod tests { #[test] fn databricks_v2_routes_by_model_family() { + use DatabricksV2Route::{AnthropicMessages, MlflowChatCompletions, OpenAiResponses}; for (model, route, path) in [ + // OpenAI-shaped: the gpt family plus the GPT-5 code names. ( "databricks-gpt-5-5", - DatabricksV2Route::OpenAiResponses, + OpenAiResponses, + "/ai-gateway/openai/v1/responses", + ), + ("gpt-4o", OpenAiResponses, "/ai-gateway/openai/v1/responses"), + // The intentional dashless `gpt5` spelling still routes to OpenAI. + ("gpt5", OpenAiResponses, "/ai-gateway/openai/v1/responses"), + ( + "databricks-gpt-5-6-luna", + OpenAiResponses, + "/ai-gateway/openai/v1/responses", + ), + ( + "databricks-gpt-5-6-sol", + OpenAiResponses, + "/ai-gateway/openai/v1/responses", + ), + ( + "databricks-terra", + OpenAiResponses, "/ai-gateway/openai/v1/responses", ), + // Anthropic-shaped: the claude prefix, the family names, and the + // release code names — each must reach the cache-capable route even + // when the endpoint name omits the literal "claude". ( "databricks-claude-opus-4-7", - DatabricksV2Route::AnthropicMessages, + AnthropicMessages, "/ai-gateway/anthropic/v1/messages", ), + ( + "goose-opus-5", + AnthropicMessages, + "/ai-gateway/anthropic/v1/messages", + ), + ( + "databricks-sonnet-5", + AnthropicMessages, + "/ai-gateway/anthropic/v1/messages", + ), + ( + "databricks-haiku-4-5", + AnthropicMessages, + "/ai-gateway/anthropic/v1/messages", + ), + ( + "databricks-mythos-5", + AnthropicMessages, + "/ai-gateway/anthropic/v1/messages", + ), + ( + "databricks-fable-5", + AnthropicMessages, + "/ai-gateway/anthropic/v1/messages", + ), + // Case-insensitive. + ( + "Databricks-Claude-Opus-5", + AnthropicMessages, + "/ai-gateway/anthropic/v1/messages", + ), + // Unrecognised names still fall through to the MLflow chat route. ( "custom-tool-model", - DatabricksV2Route::MlflowChatCompletions, + MlflowChatCompletions, + "/ai-gateway/mlflow/v1/chat/completions", + ), + ( + "databricks-gemini-3-pro", + MlflowChatCompletions, + "/ai-gateway/mlflow/v1/chat/completions", + ), + // Collision guard: short code names must match only as whole + // segments, never as substrings of an unrelated custom alias. + // Each of these embeds a marker (`sol`, `terra`, `opus`) mid-word + // and must stay on the MLflow fallback, not adopt a wire its + // backend can't parse. + ( + "consolidated-llama", + MlflowChatCompletions, + "/ai-gateway/mlflow/v1/chat/completions", + ), + ( + "terraform-coder", + MlflowChatCompletions, + "/ai-gateway/mlflow/v1/chat/completions", + ), + ( + "corpus-reranker", + MlflowChatCompletions, + "/ai-gateway/mlflow/v1/chat/completions", + ), + ( + "octopus-model", + MlflowChatCompletions, "/ai-gateway/mlflow/v1/chat/completions", ), ] { From 4555899ab2bfc7aa47b22dd872253f3704091782 Mon Sep 17 00:00:00 2001 From: klopez4212 Date: Wed, 29 Jul 2026 16:10:51 +0100 Subject: [PATCH 05/32] Polish mobile navigation and menus (#3486) ## Summary - Add a shared footer fade behind the floating tabs on Home, Activity, and Search. - Use a shared anchored popover for Activity filters and section actions, with working section move controls. - Polish message grouping/press states and remove the initial Search back button. Screenshot 2026-07-29 at 08 49 37 ### Testing - `flutter analyze` - `flutter test` - Release build installed and checked on a connected iPhone ### Screenshots A real-device Activity baseline showing the original solid footer is attached in a PR comment. The updated review build was checked on the connected iPhone. --------- Signed-off-by: kenny lopez --- .../lib/features/activity/activity_page.dart | 5 +- .../activity_page/header_actions.dart | 8 +- .../channel_detail_page/message_bubble.dart | 286 ++++++++-------- .../lib/features/channels/channels_page.dart | 2 + .../features/channels/channels_page/body.dart | 5 +- .../channels/channels_page/sections.dart | 110 ++++--- .../features/channels/thread_detail_page.dart | 306 +++++++++--------- mobile/lib/features/home/home_page.dart | 15 +- mobile/lib/features/search/search_page.dart | 12 +- .../widgets/anchored_popover_menu.dart} | 60 ++-- .../lib/shared/widgets/frosted_scaffold.dart | 6 + .../widgets/mobile_tab_footer_backdrop.dart | 61 ++++ .../features/activity/activity_page_test.dart | 12 + .../features/channels/channels_page_test.dart | 68 ++++ .../features/search/search_page_test.dart | 8 +- .../mobile_tab_footer_backdrop_test.dart | 23 ++ 16 files changed, 614 insertions(+), 373 deletions(-) rename mobile/lib/{features/activity/activity_page/popover_menu.dart => shared/widgets/anchored_popover_menu.dart} (79%) create mode 100644 mobile/lib/shared/widgets/mobile_tab_footer_backdrop.dart create mode 100644 mobile/test/shared/widgets/mobile_tab_footer_backdrop_test.dart diff --git a/mobile/lib/features/activity/activity_page.dart b/mobile/lib/features/activity/activity_page.dart index d5ea8d1c53..db39850481 100644 --- a/mobile/lib/features/activity/activity_page.dart +++ b/mobile/lib/features/activity/activity_page.dart @@ -1,6 +1,4 @@ import 'dart:async'; -import 'dart:math' as math; -import 'dart:ui' show SemanticsRole; import 'package:flutter/material.dart'; import 'package:flutter_hooks/flutter_hooks.dart'; @@ -11,6 +9,7 @@ import '../../shared/relay/relay.dart'; import '../../shared/theme/theme.dart'; import '../../shared/utils/string_utils.dart'; import '../../shared/widgets/avatar_image.dart'; +import '../../shared/widgets/anchored_popover_menu.dart'; import '../../shared/widgets/buzz_loading_indicator.dart'; import '../../shared/widgets/frosted_app_bar.dart'; import '../../shared/widgets/frosted_scaffold.dart'; @@ -34,7 +33,6 @@ import 'reminders_provider.dart'; part 'activity_page/header_actions.dart'; part 'activity_page/inbox_row.dart'; part 'activity_page/lists.dart'; -part 'activity_page/popover_menu.dart'; part 'activity_page/status_views.dart'; /// Conversation-oriented Activity inbox. @@ -284,6 +282,7 @@ class ActivityPage extends HookConsumerWidget { } return FrostedScaffold( + backgroundColor: Colors.transparent, appBar: FrostedAppBar( gradient: context.appColors.topSectionGradient, automaticallyImplyLeading: false, diff --git a/mobile/lib/features/activity/activity_page/header_actions.dart b/mobile/lib/features/activity/activity_page/header_actions.dart index 1aab78e622..56592e2e69 100644 --- a/mobile/lib/features/activity/activity_page/header_actions.dart +++ b/mobile/lib/features/activity/activity_page/header_actions.dart @@ -33,10 +33,10 @@ class _FilterMenuButton extends StatelessWidget { key: const ValueKey('activity-filter-menu'), borderRadius: BorderRadius.circular(Radii.md), onTap: () async { - final selected = await _showActivityPopover( + final selected = await showAnchoredPopover( context: buttonContext, width: 240, - alignment: _ActivityPopoverAlignment.start, + alignment: AnchoredPopoverAlignment.start, offset: const Offset(0, Grid.half), menuPadding: const EdgeInsets.symmetric(vertical: Grid.half), color: context.colors.surface.withValues(alpha: 0.98), @@ -179,10 +179,10 @@ class _InboxOptionsButton extends StatelessWidget { tooltip: 'Activity options', icon: const Icon(LucideIcons.ellipsis, size: 20), onPressed: () async { - final selected = await _showActivityPopover( + final selected = await showAnchoredPopover( context: buttonContext, width: 216, - alignment: _ActivityPopoverAlignment.end, + alignment: AnchoredPopoverAlignment.end, color: context.colors.surface, elevation: 4, shadowColor: context.colors.shadow.withValues(alpha: 0.18), diff --git a/mobile/lib/features/channels/channel_detail_page/message_bubble.dart b/mobile/lib/features/channels/channel_detail_page/message_bubble.dart index 4a49c9b210..5c428fc67b 100644 --- a/mobile/lib/features/channels/channel_detail_page/message_bubble.dart +++ b/mobile/lib/features/channels/channel_detail_page/message_bubble.dart @@ -52,156 +52,166 @@ class _MessageBubble extends ConsumerWidget { } } - return Material( - color: Colors.transparent, - borderRadius: BorderRadius.circular(Radii.md), - // The media carousel intentionally continues through the list's trailing - // gutter. InkWell still clips its ink to [borderRadius], while leaving - // overflowing message content visible. - clipBehavior: Clip.none, - child: InkWell( - key: ValueKey('message-row-${message.id}'), + return Padding( + padding: EdgeInsets.only(top: showAuthor ? Grid.xs : 0), + child: Material( + color: Colors.transparent, borderRadius: BorderRadius.circular(Radii.md), - highlightColor: context.colors.primary.withValues(alpha: 0.1), - onLongPress: () => showMessageActions( - context: context, - ref: ref, - message: message, - channelId: currentChannelId, - canManageMessage: canManageMessage, - allMessages: allMessages, - currentPubkey: currentPubkey, - isMember: isMember, - isArchived: isArchived, - ), - child: Padding( - padding: EdgeInsets.only( - top: showAuthor ? Grid.xs : Grid.xxs, - bottom: showAuthor ? 0 : Grid.xxs, + // The media carousel intentionally continues through the list's trailing + // gutter. InkWell still clips its ink to [borderRadius], while leaving + // overflowing message content visible. + clipBehavior: Clip.none, + child: InkWell( + key: ValueKey('message-row-${message.id}'), + borderRadius: BorderRadius.circular(Radii.md), + highlightColor: context.colors.primary.withValues(alpha: 0.1), + onLongPress: () => showMessageActions( + context: context, + ref: ref, + message: message, + channelId: currentChannelId, + canManageMessage: canManageMessage, + allMessages: allMessages, + currentPubkey: currentPubkey, + isMember: isMember, + isArchived: isArchived, ), - child: Row( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - if (showAuthor) - GestureDetector( - onTap: () => showUserProfileSheet(context, message.pubkey), - child: _UserAvatar(profile: profile, pubkey: message.pubkey), - ) - else - const SizedBox(width: messageAvatarSize), - const SizedBox(width: messageAvatarContentGap), - Expanded( - child: Transform.translate( - offset: Offset(0, showAuthor ? -Grid.quarter : 0), - child: Column( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - if (showAuthor) - Padding( - padding: const EdgeInsets.only(bottom: Grid.quarter), - child: Row( - children: [ - Expanded( - child: MessageAuthorMeta( - displayName: displayName, - username: messageUsernameLabel(profile), - timestamp: formatMessageTime( - message.createdAt, - ), - nameColor: context.colors.onSurface, - metadataColor: - context.colors.onSurfaceVariant, - onAuthorTap: () => showUserProfileSheet( - context, - message.pubkey, - ), - displayNameKey: ValueKey( - 'message-author-${message.id}', - ), - usernameKey: ValueKey( - 'message-username-${message.id}', - ), - timestampKey: ValueKey( - 'message-timestamp-${message.id}', + child: Padding( + padding: EdgeInsets.only( + top: showAuthor ? 0 : Grid.xxs, + bottom: showAuthor ? 0 : Grid.xxs, + ), + child: Row( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + if (showAuthor) + GestureDetector( + onTap: () => showUserProfileSheet(context, message.pubkey), + child: _UserAvatar( + profile: profile, + pubkey: message.pubkey, + ), + ) + else + const SizedBox(width: messageAvatarSize), + const SizedBox(width: messageAvatarContentGap), + Expanded( + child: Padding( + padding: EdgeInsets.only(top: showAuthor ? Grid.half : 0), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + if (showAuthor) + Padding( + padding: const EdgeInsets.only( + bottom: Grid.quarter, + ), + child: Row( + children: [ + Expanded( + child: MessageAuthorMeta( + displayName: displayName, + username: messageUsernameLabel(profile), + timestamp: formatMessageTime( + message.createdAt, + ), + nameColor: context.colors.onSurface, + metadataColor: + context.colors.onSurfaceVariant, + onAuthorTap: () => showUserProfileSheet( + context, + message.pubkey, + ), + displayNameKey: ValueKey( + 'message-author-${message.id}', + ), + usernameKey: ValueKey( + 'message-username-${message.id}', + ), + timestampKey: ValueKey( + 'message-timestamp-${message.id}', + ), ), ), - ), - if (message.edited) ...[ - const SizedBox(width: Grid.half), - Text( - '(edited)', - style: context.textTheme.labelSmall?.copyWith( - color: context.colors.onSurfaceVariant, - fontStyle: FontStyle.italic, + if (message.edited) ...[ + const SizedBox(width: Grid.half), + Text( + '(edited)', + style: context.textTheme.labelSmall + ?.copyWith( + color: + context.colors.onSurfaceVariant, + fontStyle: FontStyle.italic, + ), ), - ), + ], ], - ], + ), ), - ), - MessageContent( - content: message.content, - mentionNames: mentionNames, - agentMentionPubkeys: agentMentionPubkeys, - channelNames: channelNames, - tags: message.tags, - baseStyle: messageBodyTextStyle.copyWith( - color: context.colors.onSurface, - ), - mediaCarouselTrailingOverflow: Grid.gutter, - onMediaReply: allMessages == null - ? null - : () { - if (!context.mounted) return; - Navigator.of(context).push( - MaterialPageRoute( - builder: (_) => ThreadDetailPage( - threadHead: message, - allMessages: allMessages!, - channelId: currentChannelId, - currentPubkey: currentPubkey, - isMember: isMember, - isArchived: isArchived, + MessageContent( + content: message.content, + mentionNames: mentionNames, + agentMentionPubkeys: agentMentionPubkeys, + channelNames: channelNames, + tags: message.tags, + baseStyle: messageBodyTextStyle.copyWith( + color: context.colors.onSurface, + ), + mediaCarouselTrailingOverflow: Grid.gutter, + onMediaReply: allMessages == null + ? null + : () { + if (!context.mounted) return; + Navigator.of(context).push( + MaterialPageRoute( + builder: (_) => ThreadDetailPage( + threadHead: message, + allMessages: allMessages!, + channelId: currentChannelId, + currentPubkey: currentPubkey, + isMember: isMember, + isArchived: isArchived, + ), ), - ), - ); - }, - onMediaMore: (viewerContext, imageUrl) => - showImageActions( - context: viewerContext, + ); + }, + onMediaMore: (viewerContext, imageUrl) => + showImageActions( + context: viewerContext, + ref: ref, + message: message, + channelId: currentChannelId, + imageUrl: imageUrl, + canManageMessage: canManageMessage, + onDeleted: () { + if (viewerContext.mounted) { + Navigator.of(viewerContext).maybePop(); + } + }, + ), + onChannelTap: (channelId) { + openChannelLink( + context: context, ref: ref, - message: message, - channelId: currentChannelId, - imageUrl: imageUrl, - canManageMessage: canManageMessage, - onDeleted: () { - if (viewerContext.mounted) { - Navigator.of(viewerContext).maybePop(); - } - }, - ), - onChannelTap: (channelId) { - openChannelLink( - context: context, - ref: ref, - channelId: channelId, - currentChannelId: currentChannelId, - ); - }, - onMentionTap: (pubkey) => - showUserProfileSheet(context, pubkey), - ), - if (message.reactions.isNotEmpty) - ReactionRow( - reactions: message.reactions, - onToggle: (emoji) => - toggleReaction(ref, message, emoji), + channelId: channelId, + currentChannelId: currentChannelId, + ); + }, + onMentionTap: (pubkey) => + showUserProfileSheet(context, pubkey), ), - ], + if (message.reactions.isNotEmpty) + ReactionRow( + reactions: message.reactions, + onToggle: (emoji) => + toggleReaction(ref, message, emoji), + ), + ], + ), ), ), - ), - ], + ], + ), ), ), ), diff --git a/mobile/lib/features/channels/channels_page.dart b/mobile/lib/features/channels/channels_page.dart index 8edd056e76..ba5d4ebf9d 100644 --- a/mobile/lib/features/channels/channels_page.dart +++ b/mobile/lib/features/channels/channels_page.dart @@ -14,6 +14,7 @@ import '../../shared/community/community_icon_provider.dart'; import '../../shared/relay/relay.dart'; import '../../shared/theme/theme.dart'; import '../../shared/widgets/avatar_image.dart'; +import '../../shared/widgets/anchored_popover_menu.dart'; import '../../shared/widgets/buzz_loading_indicator.dart'; import '../../shared/widgets/frosted_app_bar.dart'; import '../../shared/widgets/frosted_scaffold.dart'; @@ -226,6 +227,7 @@ class ChannelsPage extends HookConsumerWidget { }, [isReconnectingWithContent]); return FrostedScaffold( + backgroundColor: Colors.transparent, appBar: FrostedAppBar( horizontalInset: _kTopSectionInset, // Under a Buzz theme the community + account avatar strip carries the diff --git a/mobile/lib/features/channels/channels_page/body.dart b/mobile/lib/features/channels/channels_page/body.dart index 2e0a570f41..9f0b1dd4a7 100644 --- a/mobile/lib/features/channels/channels_page/body.dart +++ b/mobile/lib/features/channels/channels_page/body.dart @@ -189,7 +189,10 @@ class _SliverChannelsList extends HookConsumerWidget { } return SliverPadding( - padding: const EdgeInsets.only(top: Grid.xxs, bottom: 80), + padding: EdgeInsets.only( + top: Grid.xxs, + bottom: MediaQuery.paddingOf(context).bottom, + ), sliver: SliverList.list( children: [ if (visibleChannels.isEmpty) diff --git a/mobile/lib/features/channels/channels_page/sections.dart b/mobile/lib/features/channels/channels_page/sections.dart index 63bd5db9d9..f9fe5453d7 100644 --- a/mobile/lib/features/channels/channels_page/sections.dart +++ b/mobile/lib/features/channels/channels_page/sections.dart @@ -152,56 +152,72 @@ class _CustomSectionHeader extends ConsumerWidget { ), ), const SizedBox(width: _kChannelLabelGap), - Text( - section.name, - style: contentListTitleTextStyle.copyWith( - color: sectionColor, - fontWeight: FontWeight.w600, + Expanded( + child: Text( + section.name, + maxLines: 1, + overflow: TextOverflow.ellipsis, + style: contentListTitleTextStyle.copyWith( + color: sectionColor, + fontWeight: FontWeight.w600, + ), ), ), - const Spacer(), - GestureDetector( - onTapUp: (details) async { - final overlay = - Overlay.of(context).context.findRenderObject()! - as RenderBox; - final position = RelativeRect.fromRect( - details.globalPosition & Size.zero, - Offset.zero & overlay.size, - ); - final value = await showMenu( - context: context, - position: position, - items: [ - const PopupMenuItem(value: 'rename', child: Text('Rename')), - PopupMenuItem( - value: 'move_up', - enabled: !isFirst, - child: const Text('Move Up'), - ), - PopupMenuItem( - value: 'move_down', - enabled: !isLast, - child: const Text('Move Down'), + Builder( + builder: (buttonContext) => IconButton( + key: ValueKey('section-menu-${section.id}'), + tooltip: '${section.name} options', + visualDensity: VisualDensity.compact, + icon: Icon( + LucideIcons.ellipsisVertical, + size: _kChannelIconSize, + color: sectionColor, + ), + onPressed: () async { + final value = await showAnchoredPopover( + context: buttonContext, + width: 216, + alignment: AnchoredPopoverAlignment.end, + color: context.colors.surface, + elevation: 4, + shadowColor: context.colors.shadow.withValues(alpha: 0.18), + shape: RoundedRectangleBorder( + borderRadius: BorderRadius.circular(Radii.md), + side: BorderSide(color: context.colors.outline), ), - const PopupMenuItem(value: 'delete', child: Text('Delete')), - ], - ); - switch (value) { - case 'rename': - onRename(); - case 'move_up': - onMoveUp(); - case 'move_down': - onMoveDown(); - case 'delete': - onDelete(); - } - }, - child: Icon( - LucideIcons.ellipsisVertical, - size: _kChannelIconSize, - color: sectionColor, + surfaceKey: ValueKey('section-popover-${section.id}'), + items: [ + const PopupMenuItem( + value: 'rename', + child: Text('Rename'), + ), + PopupMenuItem( + value: 'move_up', + enabled: !isFirst, + child: const Text('Move Up'), + ), + PopupMenuItem( + value: 'move_down', + enabled: !isLast, + child: const Text('Move Down'), + ), + const PopupMenuItem( + value: 'delete', + child: Text('Delete'), + ), + ], + ); + switch (value) { + case 'rename': + onRename(); + case 'move_up': + onMoveUp(); + case 'move_down': + onMoveDown(); + case 'delete': + onDelete(); + } + }, ), ), const SizedBox(width: Grid.quarter), diff --git a/mobile/lib/features/channels/thread_detail_page.dart b/mobile/lib/features/channels/thread_detail_page.dart index 167b111f4f..e8081f42fe 100644 --- a/mobile/lib/features/channels/thread_detail_page.dart +++ b/mobile/lib/features/channels/thread_detail_page.dart @@ -502,168 +502,172 @@ class _ThreadMessage extends ConsumerWidget { } } - return DecoratedBox( - key: ValueKey('thread-message-${message.id}'), - decoration: BoxDecoration( - color: isHighlighted - ? context.colors.primary.withValues(alpha: 0.12) - : Colors.transparent, - borderRadius: BorderRadius.circular(Radii.md), - ), - child: Material( - color: Colors.transparent, - borderRadius: BorderRadius.circular(Radii.md), - // The media carousel intentionally continues through the list's - // trailing gutter. InkWell still clips its ink to [borderRadius], - // while leaving overflowing message content visible. - clipBehavior: Clip.none, - child: InkWell( - key: ValueKey('thread-message-row-${message.id}'), + return Padding( + padding: EdgeInsets.only(top: showAuthor ? Grid.xs : 0), + child: DecoratedBox( + key: ValueKey('thread-message-${message.id}'), + decoration: BoxDecoration( + color: isHighlighted + ? context.colors.primary.withValues(alpha: 0.12) + : Colors.transparent, borderRadius: BorderRadius.circular(Radii.md), - highlightColor: context.colors.primary.withValues(alpha: 0.1), - onLongPress: () => showMessageActions( - context: context, - ref: ref, - message: message, - channelId: channelId, - canManageMessage: canManageMessage, - allMessages: allMessages, - currentPubkey: currentPubkey, - isMember: isMember, - isArchived: isArchived, - ), - child: Padding( - padding: EdgeInsets.only( - top: showAuthor ? Grid.xs : Grid.xxs, - bottom: showAuthor ? 0 : Grid.xxs, + ), + child: Material( + color: Colors.transparent, + borderRadius: BorderRadius.circular(Radii.md), + // The media carousel intentionally continues through the list's + // trailing gutter. InkWell still clips its ink to [borderRadius], + // while leaving overflowing message content visible. + clipBehavior: Clip.none, + child: InkWell( + key: ValueKey('thread-message-row-${message.id}'), + borderRadius: BorderRadius.circular(Radii.md), + highlightColor: context.colors.primary.withValues(alpha: 0.1), + onLongPress: () => showMessageActions( + context: context, + ref: ref, + message: message, + channelId: channelId, + canManageMessage: canManageMessage, + allMessages: allMessages, + currentPubkey: currentPubkey, + isMember: isMember, + isArchived: isArchived, ), - child: Row( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - if (showAuthor) - GestureDetector( - onTap: () => showUserProfileSheet(context, message.pubkey), - child: _Avatar(profile: profile, pubkey: message.pubkey), - ) - else - const SizedBox(width: messageAvatarSize), - const SizedBox(width: messageAvatarContentGap), - Expanded( - child: Transform.translate( - offset: Offset(0, showAuthor ? -Grid.quarter : 0), - child: Column( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - if (showAuthor) - Padding( - padding: const EdgeInsets.only( - bottom: Grid.quarter, - ), - child: Row( - children: [ - Expanded( - child: MessageAuthorMeta( - displayName: displayName, - username: messageUsernameLabel(profile), - timestamp: formatMessageTime( - message.createdAt, - ), - nameColor: context.colors.onSurface, - metadataColor: - context.colors.onSurfaceVariant, - onAuthorTap: () => showUserProfileSheet( - context, - message.pubkey, - ), - displayNameKey: ValueKey( - 'thread-message-author-${message.id}', - ), - usernameKey: ValueKey( - 'thread-message-username-${message.id}', - ), - timestampKey: ValueKey( - 'thread-message-timestamp-${message.id}', + child: Padding( + padding: EdgeInsets.only( + top: showAuthor ? 0 : Grid.xxs, + bottom: showAuthor ? 0 : Grid.xxs, + ), + child: Row( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + if (showAuthor) + GestureDetector( + onTap: () => + showUserProfileSheet(context, message.pubkey), + child: _Avatar(profile: profile, pubkey: message.pubkey), + ) + else + const SizedBox(width: messageAvatarSize), + const SizedBox(width: messageAvatarContentGap), + Expanded( + child: Padding( + padding: EdgeInsets.only(top: showAuthor ? Grid.half : 0), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + if (showAuthor) + Padding( + padding: const EdgeInsets.only( + bottom: Grid.quarter, + ), + child: Row( + children: [ + Expanded( + child: MessageAuthorMeta( + displayName: displayName, + username: messageUsernameLabel(profile), + timestamp: formatMessageTime( + message.createdAt, + ), + nameColor: context.colors.onSurface, + metadataColor: + context.colors.onSurfaceVariant, + onAuthorTap: () => showUserProfileSheet( + context, + message.pubkey, + ), + displayNameKey: ValueKey( + 'thread-message-author-${message.id}', + ), + usernameKey: ValueKey( + 'thread-message-username-${message.id}', + ), + timestampKey: ValueKey( + 'thread-message-timestamp-${message.id}', + ), ), ), - ), - if (message.edited) ...[ - const SizedBox(width: Grid.half), - Text( - '(edited)', - style: context.textTheme.labelSmall - ?.copyWith( - color: - context.colors.onSurfaceVariant, - fontStyle: FontStyle.italic, - ), - ), + if (message.edited) ...[ + const SizedBox(width: Grid.half), + Text( + '(edited)', + style: context.textTheme.labelSmall + ?.copyWith( + color: + context.colors.onSurfaceVariant, + fontStyle: FontStyle.italic, + ), + ), + ], ], - ], + ), ), - ), - MessageContent( - content: message.content, - mentionNames: mentionNames, - agentMentionPubkeys: agentMentionPubkeys, - channelNames: channelNames, - tags: message.tags, - baseStyle: messageBodyTextStyle.copyWith( - color: context.colors.onSurface, - ), - mediaCarouselTrailingOverflow: Grid.gutter, - onMediaReply: allMessages == null - ? null - : () { - if (!context.mounted) return; - Navigator.of(context).push( - MaterialPageRoute( - builder: (_) => ThreadDetailPage( - threadHead: message, - allMessages: allMessages!, - channelId: channelId, - currentPubkey: currentPubkey, - isMember: isMember, - isArchived: isArchived, + MessageContent( + content: message.content, + mentionNames: mentionNames, + agentMentionPubkeys: agentMentionPubkeys, + channelNames: channelNames, + tags: message.tags, + baseStyle: messageBodyTextStyle.copyWith( + color: context.colors.onSurface, + ), + mediaCarouselTrailingOverflow: Grid.gutter, + onMediaReply: allMessages == null + ? null + : () { + if (!context.mounted) return; + Navigator.of(context).push( + MaterialPageRoute( + builder: (_) => ThreadDetailPage( + threadHead: message, + allMessages: allMessages!, + channelId: channelId, + currentPubkey: currentPubkey, + isMember: isMember, + isArchived: isArchived, + ), ), - ), - ); - }, - onMediaMore: (viewerContext, imageUrl) => - showImageActions( - context: viewerContext, + ); + }, + onMediaMore: (viewerContext, imageUrl) => + showImageActions( + context: viewerContext, + ref: ref, + message: message, + channelId: channelId, + imageUrl: imageUrl, + canManageMessage: canManageMessage, + onDeleted: () { + if (viewerContext.mounted) { + Navigator.of(viewerContext).maybePop(); + } + }, + ), + onChannelTap: (targetChannelId) { + openChannelLink( + context: context, ref: ref, - message: message, - channelId: channelId, - imageUrl: imageUrl, - canManageMessage: canManageMessage, - onDeleted: () { - if (viewerContext.mounted) { - Navigator.of(viewerContext).maybePop(); - } - }, - ), - onChannelTap: (targetChannelId) { - openChannelLink( - context: context, - ref: ref, - channelId: targetChannelId, - currentChannelId: channelId, - ); - }, - onMentionTap: (pubkey) => - showUserProfileSheet(context, pubkey), - ), - if (message.reactions.isNotEmpty) - ReactionRow( - reactions: message.reactions, - onToggle: (emoji) => - toggleReaction(ref, message, emoji), + channelId: targetChannelId, + currentChannelId: channelId, + ); + }, + onMentionTap: (pubkey) => + showUserProfileSheet(context, pubkey), ), - ], + if (message.reactions.isNotEmpty) + ReactionRow( + reactions: message.reactions, + onToggle: (emoji) => + toggleReaction(ref, message, emoji), + ), + ], + ), ), ), - ), - ], + ], + ), ), ), ), diff --git a/mobile/lib/features/home/home_page.dart b/mobile/lib/features/home/home_page.dart index 7c1e5792f9..a40e317469 100644 --- a/mobile/lib/features/home/home_page.dart +++ b/mobile/lib/features/home/home_page.dart @@ -8,6 +8,7 @@ import 'package:hooks_riverpod/hooks_riverpod.dart'; import 'package:lucide_icons_flutter/lucide_icons.dart'; import '../../shared/theme/theme.dart'; +import '../../shared/widgets/mobile_tab_footer_backdrop.dart'; import '../activity/activity_page.dart'; import '../channels/channels_page.dart'; import '../search/search_page.dart'; @@ -17,12 +18,12 @@ class HomePage extends HookConsumerWidget { final WidgetBuilder settingsPageBuilder; - static const double _tabBarHeight = 56; + static const double _tabBarHeight = mobileTabBarHeight; static const double _tabBarRadius = _tabBarHeight / 2; static const double _tabBarInnerInset = Grid.half; static const double _selectedTabRadius = (_tabBarHeight - (_tabBarInnerInset * 2)) / 2; - static const double _tabBarBottomGap = Grid.twelve; + static const double _tabBarBottomGap = mobileTabBarBottomGap; static const double _tabBarHorizontalMargin = Grid.gutter; static const double _tabDestinationHorizontalPadding = Grid.sm; static const double _tabIconSize = 22; @@ -63,6 +64,7 @@ class HomePage extends HookConsumerWidget { ]; return Scaffold( + backgroundColor: Colors.transparent, // Keep the floating navigation and Home quick actions anchored while the // keyboard is visible on any tab. resizeToAvoidBottomInset: false, @@ -71,6 +73,7 @@ class HomePage extends HookConsumerWidget { child: Stack( fit: StackFit.expand, children: [ + Positioned.fill(child: ColoredBox(color: context.colors.surface)), Positioned.fill( child: MediaQuery( data: _mediaQueryWithFloatingTabBarClearance( @@ -80,6 +83,14 @@ class HomePage extends HookConsumerWidget { child: IndexedStack(index: tabIndex.value, children: pages), ), ), + Align( + alignment: Alignment.bottomCenter, + child: IgnorePointer( + child: MobileTabFooterBackdrop( + height: mobileTabFooterBackdropHeight(context), + ), + ), + ), Positioned.fill( child: ChannelQuickActionsLauncher( visible: tabIndex.value == 0, diff --git a/mobile/lib/features/search/search_page.dart b/mobile/lib/features/search/search_page.dart index 7f4dc24b45..b608b65aef 100644 --- a/mobile/lib/features/search/search_page.dart +++ b/mobile/lib/features/search/search_page.dart @@ -93,10 +93,12 @@ class SearchPage extends HookConsumerWidget { } return FrostedScaffold( + backgroundColor: Colors.transparent, // Keep the empty state centered in the page rather than the portion left // above the keyboard. resizeToAvoidBottomInset: false, appBar: FrostedAppBar( + automaticallyImplyLeading: false, gradient: context.appColors.topSectionGradient, title: const Text('Search'), titleStyle: headerTitleStyle, @@ -344,7 +346,10 @@ class _SearchBody extends ConsumerWidget { return ListView( key: const Key('search-results-list'), padding: EdgeInsets.only( - bottom: Grid.xl + MediaQuery.viewInsetsOf(context).bottom, + bottom: + Grid.xl + + MediaQuery.paddingOf(context).bottom + + MediaQuery.viewInsetsOf(context).bottom, ), children: [ if (showChannels && state.channelResults.isNotEmpty) @@ -394,7 +399,10 @@ class _RecentSearches extends StatelessWidget { return ListView( key: const Key('recent-searches-list'), padding: EdgeInsets.only( - bottom: Grid.xl + MediaQuery.viewInsetsOf(context).bottom, + bottom: + Grid.xl + + MediaQuery.paddingOf(context).bottom + + MediaQuery.viewInsetsOf(context).bottom, ), children: [ Padding( diff --git a/mobile/lib/features/activity/activity_page/popover_menu.dart b/mobile/lib/shared/widgets/anchored_popover_menu.dart similarity index 79% rename from mobile/lib/features/activity/activity_page/popover_menu.dart rename to mobile/lib/shared/widgets/anchored_popover_menu.dart index 56d0a612ec..46b50b6b00 100644 --- a/mobile/lib/features/activity/activity_page/popover_menu.dart +++ b/mobile/lib/shared/widgets/anchored_popover_menu.dart @@ -1,16 +1,30 @@ -part of '../activity_page.dart'; +import 'dart:math' as math; +import 'dart:ui' show SemanticsRole; -const _activityPopoverEnterDuration = Duration(milliseconds: 150); -const _activityPopoverExitDuration = Duration(milliseconds: 110); -const _activityPopoverStartScale = 0.96; +import 'package:flutter/material.dart'; -enum _ActivityPopoverAlignment { start, end } +import '../theme/theme.dart'; -Future _showActivityPopover({ +const _popoverEnterDuration = Duration(milliseconds: 150); +const _popoverExitDuration = Duration(milliseconds: 110); +const _popoverStartScale = 0.96; + +/// The horizontal edge a popover aligns to on its triggering control. +enum AnchoredPopoverAlignment { + /// Aligns the popover's leading edge with the trigger's leading edge. + start, + + /// Aligns the popover's trailing edge with the trigger's trailing edge. + end, +} + +/// Shows an anchored, cross-platform popup menu with the Activity controls' +/// sizing, motion, and safe-area placement. +Future showAnchoredPopover({ required BuildContext context, required List> items, required double width, - required _ActivityPopoverAlignment alignment, + required AnchoredPopoverAlignment alignment, required Color color, required ShapeBorder shape, required double elevation, @@ -36,7 +50,7 @@ Future _showActivityPopover({ final mediaQuery = MediaQuery.of(context); return navigator.push( - _ActivityPopoverRoute( + _AnchoredPopoverRoute( position: RelativeRect.fromRect(triggerRect, overlayRect), items: items, width: width, @@ -61,11 +75,11 @@ Future _showActivityPopover({ ); } -class _ActivityPopoverRoute extends PopupRoute { +class _AnchoredPopoverRoute extends PopupRoute { final RelativeRect position; final List> items; final double width; - final _ActivityPopoverAlignment alignment; + final AnchoredPopoverAlignment alignment; final Offset offset; final Color color; final ShapeBorder shape; @@ -78,7 +92,7 @@ class _ActivityPopoverRoute extends PopupRoute { final bool reducedMotion; final String _barrierLabel; - _ActivityPopoverRoute({ + _AnchoredPopoverRoute({ required this.position, required this.items, required this.width, @@ -107,11 +121,11 @@ class _ActivityPopoverRoute extends PopupRoute { @override Duration get transitionDuration => - reducedMotion ? Duration.zero : _activityPopoverEnterDuration; + reducedMotion ? Duration.zero : _popoverEnterDuration; @override Duration get reverseTransitionDuration => - reducedMotion ? Duration.zero : _activityPopoverExitDuration; + reducedMotion ? Duration.zero : _popoverExitDuration; @override Widget buildPage( @@ -123,16 +137,16 @@ class _ActivityPopoverRoute extends PopupRoute { CurveTween(curve: Curves.easeOutCubic), ); final scaleAnimation = Tween( - begin: _activityPopoverStartScale, + begin: _popoverStartScale, end: 1, ).animate(curvedAnimation); final transformOrigin = switch (alignment) { - _ActivityPopoverAlignment.start => Alignment.topLeft, - _ActivityPopoverAlignment.end => Alignment.topRight, + AnchoredPopoverAlignment.start => Alignment.topLeft, + AnchoredPopoverAlignment.end => Alignment.topRight, }; return CustomSingleChildLayout( - delegate: _ActivityPopoverLayoutDelegate( + delegate: _AnchoredPopoverLayoutDelegate( position: position, alignment: alignment, offset: offset, @@ -174,13 +188,13 @@ class _ActivityPopoverRoute extends PopupRoute { } } -class _ActivityPopoverLayoutDelegate extends SingleChildLayoutDelegate { +class _AnchoredPopoverLayoutDelegate extends SingleChildLayoutDelegate { final RelativeRect position; - final _ActivityPopoverAlignment alignment; + final AnchoredPopoverAlignment alignment; final Offset offset; final EdgeInsets screenPadding; - const _ActivityPopoverLayoutDelegate({ + const _AnchoredPopoverLayoutDelegate({ required this.position, required this.alignment, required this.offset, @@ -201,8 +215,8 @@ class _ActivityPopoverLayoutDelegate extends SingleChildLayoutDelegate { Offset getPositionForChild(Size size, Size childSize) { final anchorBottom = size.height - position.bottom; final desiredX = switch (alignment) { - _ActivityPopoverAlignment.start => position.left + offset.dx, - _ActivityPopoverAlignment.end => + AnchoredPopoverAlignment.start => position.left + offset.dx, + AnchoredPopoverAlignment.end => size.width - position.right - childSize.width + offset.dx, }; final minX = screenPadding.left; @@ -222,7 +236,7 @@ class _ActivityPopoverLayoutDelegate extends SingleChildLayoutDelegate { } @override - bool shouldRelayout(_ActivityPopoverLayoutDelegate oldDelegate) { + bool shouldRelayout(_AnchoredPopoverLayoutDelegate oldDelegate) { return position != oldDelegate.position || alignment != oldDelegate.alignment || offset != oldDelegate.offset || diff --git a/mobile/lib/shared/widgets/frosted_scaffold.dart b/mobile/lib/shared/widgets/frosted_scaffold.dart index 6b7e39fd75..fc0e2fe506 100644 --- a/mobile/lib/shared/widgets/frosted_scaffold.dart +++ b/mobile/lib/shared/widgets/frosted_scaffold.dart @@ -21,17 +21,23 @@ class FrostedScaffold extends StatelessWidget { /// Whether the body should resize when the on-screen keyboard appears. final bool? resizeToAvoidBottomInset; + /// Optional scaffold background, useful when a parent supplies a shared + /// surface behind this page. + final Color? backgroundColor; + const FrostedScaffold({ super.key, required this.appBar, required this.body, this.floatingActionButton, this.resizeToAvoidBottomInset, + this.backgroundColor, }); @override Widget build(BuildContext context) { return Scaffold( + backgroundColor: backgroundColor, resizeToAvoidBottomInset: resizeToAvoidBottomInset, floatingActionButton: floatingActionButton, body: Stack(children: [body, appBar]), diff --git a/mobile/lib/shared/widgets/mobile_tab_footer_backdrop.dart b/mobile/lib/shared/widgets/mobile_tab_footer_backdrop.dart new file mode 100644 index 0000000000..687880acb1 --- /dev/null +++ b/mobile/lib/shared/widgets/mobile_tab_footer_backdrop.dart @@ -0,0 +1,61 @@ +import 'package:flutter/material.dart'; + +import '../theme/theme.dart'; + +/// Height of the floating mobile tab bar, excluding its bottom clearance. +const mobileTabBarHeight = 56.0; + +/// Gap between the floating mobile tab bar and the bottom safe area. +const mobileTabBarBottomGap = Grid.twelve; + +/// Returns the shared footer backdrop height, including the logical safe area. +double mobileTabFooterBackdropHeight(BuildContext context) => + mobileTabBarHeight + + mobileTabBarBottomGap + + MediaQuery.paddingOf(context).bottom + + Grid.xl + + Grid.gutter; + +/// Shared fade behind the floating mobile tab bar. +class MobileTabFooterBackdrop extends StatelessWidget { + /// Vertical extent of the backdrop in logical pixels. + final double height; + + /// Gradient stop positions, from the transparent top to the opaque bottom. + final List stops; + + /// Surface-color alpha values paired with [stops]. + final List opacities; + + /// Creates a footer backdrop with the required [height]. + /// + /// Override [stops] and [opacities] together to customize the gradient. + const MobileTabFooterBackdrop({ + super.key, + required this.height, + this.stops = const [0, 0.5, 1], + this.opacities = const [0, 0.75, 1], + }) : assert(stops.length == opacities.length); + + @override + Widget build(BuildContext context) { + final surface = context.colors.surface; + return SizedBox( + height: height, + width: double.infinity, + child: DecoratedBox( + decoration: BoxDecoration( + gradient: LinearGradient( + begin: Alignment.topCenter, + end: Alignment.bottomCenter, + stops: stops, + colors: [ + for (final opacity in opacities) + surface.withValues(alpha: opacity), + ], + ), + ), + ), + ); + } +} diff --git a/mobile/test/features/activity/activity_page_test.dart b/mobile/test/features/activity/activity_page_test.dart index 19b170b52a..a0293455f2 100644 --- a/mobile/test/features/activity/activity_page_test.dart +++ b/mobile/test/features/activity/activity_page_test.dart @@ -182,6 +182,18 @@ void main() { expect(find.byTooltip('Back'), findsNothing); }); + testWidgets('keeps bottom clearance for the floating tab bar', ( + tester, + ) async { + await tester.pumpWidget(await buildTestable()); + await tester.pumpAndSettle(); + + final safeAreas = tester.widgetList(find.byType(SafeArea)); + expect(safeAreas, hasLength(1)); + expect(safeAreas.single.top, isFalse); + expect(safeAreas.single.bottom, isTrue); + }); + testWidgets('shows error view with retry button', (tester) async { await tester.pumpWidget( await buildTestable(activityNotifier: _ErrorActivityNotifier.new), diff --git a/mobile/test/features/channels/channels_page_test.dart b/mobile/test/features/channels/channels_page_test.dart index 2a624cbde7..991db3b5cd 100644 --- a/mobile/test/features/channels/channels_page_test.dart +++ b/mobile/test/features/channels/channels_page_test.dart @@ -8,6 +8,8 @@ import 'package:hooks_riverpod/misc.dart'; import 'package:lucide_icons_flutter/lucide_icons.dart'; import 'package:buzz/features/channels/channel.dart'; import 'package:buzz/features/channels/channel_management_provider.dart'; +import 'package:buzz/features/channels/channel_sections/channel_sections_provider.dart'; +import 'package:buzz/features/channels/channel_sections/channel_sections_storage.dart'; import 'package:buzz/features/channels/channels_page.dart'; import 'package:buzz/features/channels/channels_provider.dart'; import 'package:buzz/features/channels/read_state/read_state_provider.dart'; @@ -28,6 +30,7 @@ void main() { bool previewDirectory = false, double keyboardInset = 0, bool disableAnimations = false, + double bottomPadding = 0, Map communityIcons = const {}, ValueChanged? onCommunityIconLoad, TextScaler textScaler = TextScaler.noScaling, @@ -52,6 +55,7 @@ void main() { data: MediaQuery.of(context).copyWith( disableAnimations: disableAnimations, textScaler: textScaler, + padding: EdgeInsets.only(bottom: bottomPadding), viewInsets: EdgeInsets.only(bottom: keyboardInset), ), child: child!, @@ -142,6 +146,60 @@ void main() { expect(sectionTitle.style?.fontWeight, FontWeight.w600); }); + testWidgets('keeps the last channel above the floating tab bar', ( + tester, + ) async { + const footerClearance = 102.0; + await tester.pumpWidget( + buildTestable( + bottomPadding: footerClearance, + overrides: [ + channelsProvider.overrideWith(() => _FakeNotifier(testChannels)), + ], + ), + ); + await tester.pumpAndSettle(); + + final padding = tester.widget( + find.descendant( + of: find.byType(CustomScrollView), + matching: find.byType(SliverPadding), + ), + ); + expect((padding.padding as EdgeInsets).bottom, footerClearance); + }); + + testWidgets('truncates long custom section names beside the menu', ( + tester, + ) async { + tester.view.physicalSize = const Size(320, 800); + tester.view.devicePixelRatio = 1; + addTearDown(tester.view.reset); + const sectionName = 'A deliberately long custom section name for testing'; + await tester.pumpWidget( + buildTestable( + overrides: [ + channelsProvider.overrideWith(() => _FakeNotifier(testChannels)), + channelSectionsProvider.overrideWith( + () => _FakeChannelSectionsNotifier( + const ChannelSectionStore( + sections: [ + ChannelSection(id: 'section-1', name: sectionName, order: 0), + ], + ), + ), + ), + ], + ), + ); + await tester.pumpAndSettle(); + + final label = tester.widget(find.text(sectionName)); + expect(label.maxLines, 1); + expect(label.overflow, TextOverflow.ellipsis); + expect(tester.takeException(), isNull); + }); + testWidgets('aligns the top, section, row, and skeleton label columns', ( tester, ) async { @@ -1384,6 +1442,16 @@ class _FakeNotifier extends ChannelsNotifier { get observedUnreadEventsByChannel => _observedEventsByChannel; } +class _FakeChannelSectionsNotifier extends ChannelSectionsNotifier { + _FakeChannelSectionsNotifier(this._store); + + final ChannelSectionStore _store; + + @override + ChannelSectionsState build() => + ChannelSectionsState(isReady: true, store: _store, version: 1); +} + class _FakeCommunityListNotifier extends CommunityListNotifier { _FakeCommunityListNotifier(this._communities); diff --git a/mobile/test/features/search/search_page_test.dart b/mobile/test/features/search/search_page_test.dart index 52c2fb23ae..e4a576b91d 100644 --- a/mobile/test/features/search/search_page_test.dart +++ b/mobile/test/features/search/search_page_test.dart @@ -248,6 +248,7 @@ void main() { tester, ) async { const keyboardInset = 300.0; + const footerClearance = 102.0; await tester.pumpWidget( WidgetHelpers.testable( @@ -266,6 +267,7 @@ void main() { child: Builder( builder: (context) => MediaQuery( data: MediaQuery.of(context).copyWith( + padding: const EdgeInsets.only(bottom: footerClearance), viewInsets: const EdgeInsets.only(bottom: keyboardInset), ), child: const SearchPage(), @@ -282,13 +284,14 @@ void main() { ); final padding = recentSearches.padding! as EdgeInsets; - expect(padding.bottom, Grid.xl + keyboardInset); + expect(padding.bottom, Grid.xl + footerClearance + keyboardInset); }); testWidgets('keeps search results scrollable above the keyboard', ( tester, ) async { const keyboardInset = 300.0; + const footerClearance = 102.0; final state = SearchState( query: 'general', channelResults: [ @@ -318,6 +321,7 @@ void main() { child: Builder( builder: (context) => MediaQuery( data: MediaQuery.of(context).copyWith( + padding: const EdgeInsets.only(bottom: footerClearance), viewInsets: const EdgeInsets.only(bottom: keyboardInset), ), child: const SearchPage(), @@ -332,7 +336,7 @@ void main() { ); final padding = results.padding! as EdgeInsets; - expect(padding.bottom, Grid.xl + keyboardInset); + expect(padding.bottom, Grid.xl + footerClearance + keyboardInset); }); testWidgets('keeps no-results feedback above the keyboard', (tester) async { diff --git a/mobile/test/shared/widgets/mobile_tab_footer_backdrop_test.dart b/mobile/test/shared/widgets/mobile_tab_footer_backdrop_test.dart new file mode 100644 index 0000000000..9e54177d61 --- /dev/null +++ b/mobile/test/shared/widgets/mobile_tab_footer_backdrop_test.dart @@ -0,0 +1,23 @@ +import 'package:buzz/shared/widgets/mobile_tab_footer_backdrop.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter_test/flutter_test.dart'; + +void main() { + testWidgets('uses the logical bottom safe-area inset', (tester) async { + double? height; + + await tester.pumpWidget( + MediaQuery( + data: const MediaQueryData(padding: EdgeInsets.only(bottom: 34)), + child: Builder( + builder: (context) { + height = mobileTabFooterBackdropHeight(context); + return const SizedBox(); + }, + ), + ), + ); + + expect(height, 170); + }); +} From f7a3988ba13b590d9a55a7e8413fc3fb5ffbef18 Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 09:13:54 -0600 Subject: [PATCH 06/32] fix(desktop): preserve shared agent fidelity (#3553) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Fixes two distinct fidelity failures in direct agent sharing: - The sender now puts the same effective avatar shown on the agent card into People-share and file-export snapshot PNGs, including profile/kind:0 fallback avatars. - The importer now persists the visible PNG body as the portable avatar instead of ignoring it in favor of sender-local manifest references. - Export materializes inherited runtime, provider, and model identifiers verbatim, while preserving explicit definition values. It does not translate or substitute configuration for a different recipient setup. - Sharing waits for a profile-only fallback avatar query, preventing an early-click race. The PNG import path keeps the existing safety invariant: decode is capped at 2048×2048 / 32 MiB and re-encoded avatars above the 2 MiB inline limit fall back to the manifest reference. The exact transparent 1×1 no-avatar placeholder is ignored. The original Tyler↔Wes screenshot demonstrates both stages: Wren's attachment had an avatar that disappeared after **Add agent** (receiver/import failure), while Pinky's attachment was already blank (sender/projection failure). ### Related issue N/A — reported and traced in the linked Buzz conversation. ### Testing - `cargo test --manifest-path desktop/src-tauri/Cargo.toml commands::personas::snapshot` — 57 passed - `pnpm exec tsc --noEmit` - Biome check on changed frontend/E2E files - Pre-push hooks: - desktop check - desktop tests - desktop Tauri tests — 1853 passed, 14 ignored - file-size ratchet The People-share E2E regression asserts that a profile-only avatar reaches `avatarPngDataUrl` in the real encode command payload. --------- Signed-off-by: Wes Co-authored-by: Carl --- .../src/commands/personas/snapshot.rs | 36 +++ .../personas/snapshot/fidelity_tests.rs | 210 ++++++++++++++++++ .../src/commands/personas/snapshot/import.rs | 10 +- desktop/src-tauri/src/managed_agents/mod.rs | 1 + .../src/managed_agents/snapshot_avatar.rs | 42 ++++ desktop/src/features/agents/ui/AgentsView.tsx | 2 + .../features/agents/ui/PersonaShareDialog.tsx | 6 +- .../agents/ui/UnifiedAgentsSection.tsx | 23 +- .../features/agents/ui/usePersonaActions.ts | 7 +- desktop/tests/e2e/agents.spec.ts | 23 ++ 10 files changed, 350 insertions(+), 10 deletions(-) create mode 100644 desktop/src-tauri/src/commands/personas/snapshot/fidelity_tests.rs create mode 100644 desktop/src-tauri/src/managed_agents/snapshot_avatar.rs diff --git a/desktop/src-tauri/src/commands/personas/snapshot.rs b/desktop/src-tauri/src/commands/personas/snapshot.rs index a3ba731875..583296dac0 100644 --- a/desktop/src-tauri/src/commands/personas/snapshot.rs +++ b/desktop/src-tauri/src/commands/personas/snapshot.rs @@ -164,6 +164,33 @@ fn parse_format_is_png(s: &str) -> Result { } } +fn materialize_portable_runtime_defaults( + record: &mut ManagedAgentRecord, + global: &crate::managed_agents::GlobalAgentConfig, +) { + if record + .model + .as_deref() + .is_none_or(|value| value.trim().is_empty()) + { + record.model = global.model.clone(); + } + if record + .provider + .as_deref() + .is_none_or(|value| value.trim().is_empty()) + { + record.provider = global.provider.clone(); + } + if record + .runtime + .as_deref() + .is_none_or(|value| value.trim().is_empty()) + { + record.runtime = global.preferred_runtime.clone(); + } +} + /// Shared production encoding path. /// /// Resolves the agent definition, validates inputs, fetches optional memory, @@ -196,6 +223,13 @@ pub(crate) async fn materialize_snapshot_bytes( let definitions = load_agent_definitions(&app)?; let (def_record, is_definition) = resolve_from_lists(&id, &instances, &definitions) .map(|(r, is_def)| (r.clone(), is_def))?; + let mut def_record = def_record; + // A snapshot is a verbatim portable copy of the effective runtime, + // provider, and model configuration, not a pointer to the sender's + // machine-wide defaults. This does not translate or substitute values + // for a different recipient setup. + let global = crate::managed_agents::load_global_agent_config(&app).unwrap_or_default(); + materialize_portable_runtime_defaults(&mut def_record, &global); let memory_pubkey = if memory_level != MemoryLevel::None { let mpk = memory_source_pubkey.as_deref().unwrap_or(""); @@ -400,6 +434,8 @@ pub async fn encode_agent_snapshot_for_send( }) } +#[cfg(test)] +mod fidelity_tests; #[cfg(test)] mod tests; diff --git a/desktop/src-tauri/src/commands/personas/snapshot/fidelity_tests.rs b/desktop/src-tauri/src/commands/personas/snapshot/fidelity_tests.rs new file mode 100644 index 0000000000..00a1457393 --- /dev/null +++ b/desktop/src-tauri/src/commands/personas/snapshot/fidelity_tests.rs @@ -0,0 +1,210 @@ +use super::import::decode_snapshot_from_bytes; +use super::*; +use crate::managed_agents::{ + agent_snapshot::{ + AgentSnapshot, AgentSnapshotDefinition, AgentSnapshotMemory, AgentSnapshotProfile, + FORMAT_DISCRIMINATOR, FORMAT_VERSION, + }, + BackendKind, ManagedAgentRecord, RespondTo, +}; +use std::collections::BTreeMap; + +fn make_definition(slug: &str) -> ManagedAgentRecord { + ManagedAgentRecord { + pubkey: String::new(), + slug: Some(slug.to_string()), + name: slug.to_string(), + display_name: None, + persona_id: None, + private_key_nsec: String::new(), + auth_tag: None, + relay_url: String::new(), + avatar_url: None, + acp_command: String::new(), + agent_command: String::new(), + agent_command_override: None, + agent_args: vec![], + mcp_command: String::new(), + turn_timeout_seconds: 0, + idle_timeout_seconds: None, + max_turn_duration_seconds: None, + parallelism: 1, + system_prompt: None, + model: None, + provider: None, + persona_source_version: None, + env_vars: BTreeMap::new(), + start_on_app_launch: false, + auto_restart_on_config_change: false, + runtime_pid: None, + backend: BackendKind::Local, + backend_agent_id: None, + provider_binary_path: None, + team_id: None, + persona_team_dir: None, + persona_name_in_team: None, + created_at: String::new(), + updated_at: String::new(), + last_started_at: None, + last_stopped_at: None, + last_exit_code: None, + last_error: None, + last_error_code: None, + respond_to: RespondTo::default(), + respond_to_allowlist: vec![], + runtime: None, + name_pool: vec![], + is_builtin: false, + is_active: false, + shared: false, + source_team: None, + source_team_persona_slug: None, + catalog_source: None, + definition_respond_to: None, + definition_respond_to_allowlist: vec![], + definition_parallelism: None, + relay_mesh: None, + } +} + +/// Build a minimal valid AgentSnapshot for import tests. +fn make_snapshot( + memory_level: MemoryLevel, + entries: Vec, +) -> AgentSnapshot { + AgentSnapshot { + format: FORMAT_DISCRIMINATOR.to_string(), + version: FORMAT_VERSION, + definition: AgentSnapshotDefinition { + name: "Test Agent".to_string(), + source_is_builtin: false, + system_prompt: Some("You are helpful.".to_string()), + runtime: None, + model: None, + provider: None, + parallelism: None, + respond_to: None, + respond_to_allowlist: vec![], + name_pool: vec![], + idle_timeout_seconds: None, + max_turn_duration_seconds: None, + }, + profile: AgentSnapshotProfile { + display_name: "Test Agent".to_string(), + about: None, + avatar_data_url: None, + avatar_url: None, + }, + memory: AgentSnapshotMemory { + level: memory_level, + entries, + }, + } +} + +// ── Portable effective configuration ───────────────────────────────────── + +#[test] +fn inherited_runtime_provider_and_model_are_materialized_for_export() { + let mut record = make_definition("wren"); + let global = crate::managed_agents::GlobalAgentConfig { + preferred_runtime: Some("goose".to_string()), + provider: Some("databricks_v2".to_string()), + model: Some("databricks-gpt-5-6-sol".to_string()), + ..Default::default() + }; + + materialize_portable_runtime_defaults(&mut record, &global); + + assert_eq!(record.runtime.as_deref(), Some("goose")); + assert_eq!(record.provider.as_deref(), Some("databricks_v2")); + assert_eq!(record.model.as_deref(), Some("databricks-gpt-5-6-sol")); +} + +#[test] +fn explicit_runtime_provider_and_model_win_over_global_defaults() { + let mut record = make_definition("wren"); + record.runtime = Some("claude".to_string()); + record.provider = Some("anthropic".to_string()); + record.model = Some("claude-opus-5".to_string()); + let global = crate::managed_agents::GlobalAgentConfig { + preferred_runtime: Some("goose".to_string()), + provider: Some("databricks_v2".to_string()), + model: Some("databricks-gpt-5-6-sol".to_string()), + ..Default::default() + }; + + materialize_portable_runtime_defaults(&mut record, &global); + + assert_eq!(record.runtime.as_deref(), Some("claude")); + assert_eq!(record.provider.as_deref(), Some("anthropic")); + assert_eq!(record.model.as_deref(), Some("claude-opus-5")); +} + +/// PNG image-body avatar overrides manifest avatar fields and all definition +/// config survives the exact production decoder. +#[test] +fn import_png_body_avatar_and_full_model_round_trip() { + use crate::managed_agents::agent_snapshot::{decode_avatar_data_url, encode_snapshot_png}; + + let mut snapshot = make_snapshot(MemoryLevel::None, vec![]); + snapshot.definition.runtime = Some("goose".to_string()); + snapshot.definition.model = Some("databricks-gpt-5-6-sol".to_string()); + snapshot.definition.provider = Some("databricks_v2".to_string()); + snapshot.profile.avatar_data_url = None; + snapshot.profile.avatar_url = Some("https://sender.invalid/avatar.png".to_string()); + + let avatar = image::DynamicImage::ImageRgba8(image::RgbaImage::from_pixel( + 4, + 3, + image::Rgba([23, 91, 177, 255]), + )); + let mut avatar_png = std::io::Cursor::new(Vec::new()); + avatar + .write_to(&mut avatar_png, image::ImageFormat::Png) + .unwrap(); + let png_bytes = encode_snapshot_png(&snapshot, Some(avatar_png.get_ref())).unwrap(); + + let decoded = decode_snapshot_from_bytes(&png_bytes).unwrap(); + assert_eq!(decoded.definition.runtime.as_deref(), Some("goose")); + assert_eq!( + decoded.definition.model.as_deref(), + Some("databricks-gpt-5-6-sol") + ); + assert_eq!( + decoded.definition.provider.as_deref(), + Some("databricks_v2") + ); + assert_eq!( + decoded.profile.avatar_url.as_deref(), + Some("https://sender.invalid/avatar.png") + ); + + let avatar_data_url = decoded + .profile + .avatar_data_url + .as_deref() + .expect("PNG image body must become the effective portable avatar"); + let avatar_bytes = decode_avatar_data_url(avatar_data_url).unwrap(); + let imported_avatar = image::load_from_memory(&avatar_bytes).unwrap(); + assert_eq!((imported_avatar.width(), imported_avatar.height()), (4, 3)); + assert_eq!( + imported_avatar.to_rgba8().get_pixel(0, 0).0, + [23, 91, 177, 255] + ); +} + +/// The transparent 1×1 no-avatar card must not override a manifest fallback. +#[test] +fn import_png_placeholder_keeps_manifest_avatar_fallback() { + use crate::managed_agents::agent_snapshot::encode_snapshot_png; + + let mut snapshot = make_snapshot(MemoryLevel::None, vec![]); + snapshot.profile.avatar_data_url = None; + snapshot.profile.avatar_url = Some("https://example.com/avatar.png".to_string()); + let png_bytes = encode_snapshot_png(&snapshot, None).unwrap(); + + let decoded = decode_snapshot_from_bytes(&png_bytes).unwrap(); + assert!(decoded.profile.avatar_data_url.is_none()); + assert_eq!(decoded.profile.avatar_url, snapshot.profile.avatar_url); +} diff --git a/desktop/src-tauri/src/commands/personas/snapshot/import.rs b/desktop/src-tauri/src/commands/personas/snapshot/import.rs index 9d7d238918..7648941d23 100644 --- a/desktop/src-tauri/src/commands/personas/snapshot/import.rs +++ b/desktop/src-tauri/src/commands/personas/snapshot/import.rs @@ -220,7 +220,15 @@ pub(crate) fn decode_snapshot_from_bytes( file_bytes.len() / (1024 * 1024) )); } - let snapshot = decode_snapshot_png(file_bytes)?; + let mut snapshot = decode_snapshot_png(file_bytes)?; + // The PNG image body is the portable avatar. It deliberately wins over + // manifest avatar fields, whose URL may only be reachable by the + // sender. A 1×1 export placeholder leaves the manifest fallback intact. + if let Some(avatar_data_url) = + crate::managed_agents::snapshot_avatar::snapshot_png_avatar_data_url(file_bytes)? + { + snapshot.profile.avatar_data_url = Some(avatar_data_url); + } if snapshot.memory.level == MemoryLevel::None && !snapshot.memory.entries.is_empty() { return Err( "Snapshot is malformed: memory.level is 'none' but entries are present." diff --git a/desktop/src-tauri/src/managed_agents/mod.rs b/desktop/src-tauri/src/managed_agents/mod.rs index b0e86f8edb..be9b07cf11 100644 --- a/desktop/src-tauri/src/managed_agents/mod.rs +++ b/desktop/src-tauri/src/managed_agents/mod.rs @@ -29,6 +29,7 @@ pub mod retention; mod runtime; mod runtime_commands; mod runtime_types; +pub(crate) mod snapshot_avatar; pub(crate) mod spawn_hash; pub(crate) mod storage; pub(crate) mod team_events; diff --git a/desktop/src-tauri/src/managed_agents/snapshot_avatar.rs b/desktop/src-tauri/src/managed_agents/snapshot_avatar.rs new file mode 100644 index 0000000000..a1044b31f4 --- /dev/null +++ b/desktop/src-tauri/src/managed_agents/snapshot_avatar.rs @@ -0,0 +1,42 @@ +use base64::{engine::general_purpose::STANDARD, Engine as _}; +use image::ImageDecoder; +use std::io::Cursor; + +const MAX_AVATAR_INLINE_BYTES: usize = 2 * 1024 * 1024; +const MAX_AVATAR_DIMENSION: u32 = 2048; +const MAX_AVATAR_DECODE_ALLOC: u64 = 32 * 1024 * 1024; + +/// Materialize a snapshot PNG's visible pixels as a bounded portable avatar. +/// The exact transparent 1×1 no-avatar placeholder and images that cannot fit +/// the persisted inline-avatar budget leave the manifest fallback intact. +pub(crate) fn snapshot_png_avatar_data_url(png_bytes: &[u8]) -> Result, String> { + let reader = image::ImageReader::with_format(Cursor::new(png_bytes), image::ImageFormat::Png); + let mut decoder = reader + .into_decoder() + .map_err(|e| format!("Failed to decode snapshot avatar: {e}"))?; + let mut limits = image::Limits::default(); + limits.max_image_width = Some(MAX_AVATAR_DIMENSION); + limits.max_image_height = Some(MAX_AVATAR_DIMENSION); + limits.max_alloc = Some(MAX_AVATAR_DECODE_ALLOC); + decoder + .set_limits(limits) + .map_err(|e| format!("Snapshot avatar exceeds safe decoding limits: {e}"))?; + let (width, height) = decoder.dimensions(); + let image = image::DynamicImage::from_decoder(decoder) + .map_err(|e| format!("Failed to decode snapshot avatar: {e}"))?; + if width == 1 && height == 1 && image.to_rgba8().get_pixel(0, 0).0 == [0, 0, 0, 0] { + return Ok(None); + } + + let mut clean_png = Vec::new(); + image + .write_to(&mut Cursor::new(&mut clean_png), image::ImageFormat::Png) + .map_err(|e| format!("Failed to encode snapshot avatar: {e}"))?; + if clean_png.len() > MAX_AVATAR_INLINE_BYTES { + return Ok(None); + } + Ok(Some(format!( + "data:image/png;base64,{}", + STANDARD.encode(clean_png) + ))) +} diff --git a/desktop/src/features/agents/ui/AgentsView.tsx b/desktop/src/features/agents/ui/AgentsView.tsx index 6e55f92dfe..f24a3c06d7 100644 --- a/desktop/src/features/agents/ui/AgentsView.tsx +++ b/desktop/src/features/agents/ui/AgentsView.tsx @@ -358,6 +358,7 @@ export function AgentsView() { )} isPending={personas.isPending} linkedAgentPubkey={personas.personaToShare.linkedAgentPubkey} + effectiveAvatarUrl={personas.personaToShare.effectiveAvatarUrl} onCatalogShareLevelChange={(shareLevel) => { const shareTarget = personas.personaToShare; if (!shareTarget) return; @@ -392,6 +393,7 @@ export function AgentsView() { personas.handleExportSnapshot( personas.personaToExportSnapshot.persona, personas.personaToExportSnapshot.linkedAgentPubkey, + personas.personaToExportSnapshot.effectiveAvatarUrl, memoryLevel, format, ); diff --git a/desktop/src/features/agents/ui/PersonaShareDialog.tsx b/desktop/src/features/agents/ui/PersonaShareDialog.tsx index af45e8f071..c641de9c70 100644 --- a/desktop/src/features/agents/ui/PersonaShareDialog.tsx +++ b/desktop/src/features/agents/ui/PersonaShareDialog.tsx @@ -54,6 +54,7 @@ type PersonaShareDialogProps = { catalogShareLevel: CatalogPersonaShareLevel; isPending: boolean; linkedAgentPubkey: string | null; + effectiveAvatarUrl: string | null; onCatalogShareLevelChange: (shareLevel: CatalogPersonaShareLevel) => void; onExport: () => void; onOpenChange: (open: boolean) => void; @@ -694,6 +695,7 @@ export function PersonaShareDialog({ catalogShareLevel, isPending, linkedAgentPubkey, + effectiveAvatarUrl, onCatalogShareLevelChange, onExport, onOpenChange, @@ -715,12 +717,12 @@ export function PersonaShareDialog({ memoryLevel: linkedAgentPubkey ? memoryLevel : "none", format: "png", memorySourcePubkey: linkedAgentPubkey, - avatarPngDataUrl: await resolveSnapshotAvatarPng(persona.avatarUrl), + avatarPngDataUrl: await resolveSnapshotAvatarPng(effectiveAvatarUrl), }), [ encodeSnapshotMutation.mutateAsync, + effectiveAvatarUrl, linkedAgentPubkey, - persona.avatarUrl, persona.id, ], ); diff --git a/desktop/src/features/agents/ui/UnifiedAgentsSection.tsx b/desktop/src/features/agents/ui/UnifiedAgentsSection.tsx index 34f9f9819f..9bbe3feef7 100644 --- a/desktop/src/features/agents/ui/UnifiedAgentsSection.tsx +++ b/desktop/src/features/agents/ui/UnifiedAgentsSection.tsx @@ -58,6 +58,7 @@ type UnifiedAgentsSectionProps = { onSharePersona: ( persona: AgentPersona, linkedAgent: ManagedAgent | undefined, + effectiveAvatarUrl: string | null, ) => void; onDeactivatePersona: (persona: AgentPersona) => void; onDeletePersona: (persona: AgentPersona) => void; @@ -157,9 +158,11 @@ export function UnifiedAgentsSection(props: UnifiedAgentsSectionProps) { const profileAgent = pickProfileAgent(group.agents); return ( ( + onSharePersona(persona, linkedAgent, effectiveAvatarUrl) + } /> - } + )} agent={profileAgent} defaultModel={defaultModel} key={group.persona.id} @@ -250,7 +255,10 @@ function AgentPersonaCard({ onStartAgent, onStartPersona, }: { - actions?: React.ReactNode; + actions?: ( + effectiveAvatarUrl: string | null, + isEffectiveAvatarLoading: boolean, + ) => React.ReactNode; agent: ManagedAgent | undefined; defaultModel: string; persona: AgentPersona; @@ -282,7 +290,10 @@ function AgentPersonaCard({ return ( (null); const [personaToExportSnapshot, setPersonaToExportSnapshot] = React.useState<{ persona: AgentPersona; linkedAgentPubkey: string | null; + effectiveAvatarUrl: string | null; } | null>(null); const [snapshotImportState, setSnapshotImportState] = React.useState<{ fileBytes: number[]; @@ -447,17 +449,20 @@ export function usePersonaActions() { function openShare( persona: AgentPersona, linkedAgent: ManagedAgent | undefined, + effectiveAvatarUrl: string | null, ) { clearFeedback("library"); setPersonaToShare({ persona, linkedAgentPubkey: linkedAgent?.pubkey ?? null, + effectiveAvatarUrl, }); } function handleExportSnapshot( persona: AgentPersona, linkedAgentPubkey: string | null, + effectiveAvatarUrl: string | null, memoryLevel: SnapshotMemoryLevel, format: SnapshotFormat, ) { @@ -469,7 +474,7 @@ export function usePersonaActions() { memoryLevel, format, memorySourcePubkey: linkedAgentPubkey, - avatarUrl: persona.avatarUrl, + avatarUrl: effectiveAvatarUrl, }, { onSuccess: (saved) => { diff --git a/desktop/tests/e2e/agents.spec.ts b/desktop/tests/e2e/agents.spec.ts index 2e7cdc9e83..13eda788b1 100644 --- a/desktop/tests/e2e/agents.spec.ts +++ b/desktop/tests/e2e/agents.spec.ts @@ -1733,6 +1733,20 @@ test("one share level selector drives both the link and send paths", async ({ }) => { await page.emulateMedia({ reducedMotion: "no-preference" }); const linkedAgentPubkey = TEST_IDENTITIES.alice.pubkey; + const profileAvatarUrl = "https://mock.relay/media/profile-only-avatar.png"; + const profileAvatarBytes = Uint8Array.from( + atob( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", + ), + (character) => character.charCodeAt(0), + ); + await page.route(profileAvatarUrl, async (route) => { + await route.fulfill({ + body: Buffer.from(profileAvatarBytes), + contentType: "image/png", + status: 200, + }); + }); await page.context().grantPermissions(["clipboard-read", "clipboard-write"]); await installMockBridge(page, { personas: [ @@ -1751,6 +1765,12 @@ test("one share level selector drives both the link and send paths", async ({ }, ], searchProfiles: [ + { + pubkey: linkedAgentPubkey, + displayName: "Animation Auditor", + avatarUrl: profileAvatarUrl, + isAgent: true, + }, { pubkey: TEST_IDENTITIES.charlie.pubkey, displayName: "Charlie", @@ -1999,6 +2019,9 @@ test("one share level selector drives both the link and send paths", async ({ expect.objectContaining({ memoryLevel: "core", memorySourcePubkey: linkedAgentPubkey, + avatarPngDataUrl: `data:image/png;base64,${Buffer.from( + profileAvatarBytes, + ).toString("base64")}`, }), expect.objectContaining({ memoryLevel: "everything", From 294c8c821de51442a8c384c0bdb66b1a10224ca0 Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 11:17:35 -0400 Subject: [PATCH 07/32] perf(desktop): move observer-feed archive and decrypt commands off main thread (#3415) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opening the agent observer feed could beachball the app. In Tauri 2, a sync (`pub fn`) command body runs on the **main thread** — only `async fn` commands run on the runtime pool. Five commands on the observer-feed open path were sync, so panel open ran SQLite I/O and secp256k1 work on the macOS main thread: | Command | Main-thread work | |---|---| | `decrypt_observer_event` | Schnorr ID + signature verify, then NIP-44 decrypt — once per frame | | `read_archived_observer_events_for_channel` | Opens the archive DB, runs the channel-index JOIN, returns up to 200 raw JSON blobs per page | | `read_unindexed_observer_rows` | Opens the DB, returns **all** not-yet-indexed kind-24200 rows in one shot | | `index_observer_channel_id` | Opens the DB, loops N upserts | | `delete_save_subscription` | Opens the DB, one delete | Eager hydration loads up to 10 pages × 200 frames on panel open, so that's up to 10 main-thread DB reads plus up to 2,000 sequential verify+decrypt calls before any scrolling. The one-shot backfill makes it worse on the first open after history accumulates: one read of every unindexed row, a decrypt per row, then a batch upsert — all on the main thread, and all proportional to archive size. The four archive commands now route their DB work through the existing `run_archive_db_task` helper (`spawn_blocking` + `open_db`), matching `list_save_subscriptions`, `read_archived_events`, and `archive_events` directly around them. `decrypt_observer_event` becomes `async fn` + `tauri::async_runtime::spawn_blocking`, with `state.signing_keys()` extracted before the spawn since `State` is not `Send` — the same pattern `sign_event` uses from #1222. No frontend changes: `invoke` is already promise-based, so the TS wrappers in `tauriArchive.ts` and `tauriObserver.ts` are unchanged. This removes the freeze, not the work. Eager hydration still takes the same wall time — the feed shows a loading state instead of blocking the UI. Batching the per-frame decrypt IPC (2,000 round-trips into one command) would cut the latency itself; that's deliberately out of scope here. Signed-off-by: Will Pfleger --- desktop/src-tauri/src/archive/mod.rs | 96 ++++++++++++---------- desktop/src-tauri/src/commands/identity.rs | 28 ++++--- 2 files changed, 69 insertions(+), 55 deletions(-) diff --git a/desktop/src-tauri/src/archive/mod.rs b/desktop/src-tauri/src/archive/mod.rs index a65b126a4d..42c6812674 100644 --- a/desktop/src-tauri/src/archive/mod.rs +++ b/desktop/src-tauri/src/archive/mod.rs @@ -483,21 +483,23 @@ pub async fn list_save_subscriptions( /// Does NOT purge already-archived event data — retention is decoupled in v1. /// GC of orphaned event rows happens in P4 purge commands, not here. #[tauri::command] -pub fn delete_save_subscription( +pub async fn delete_save_subscription( state: State<'_, AppState>, scope_type: ScopeType, scope_value: String, ) -> Result { let identity_pk = identity_pubkey(&state)?; let relay_url = relay_ws_url_with_override(&state); - let conn = open_db()?; - store::delete_save_subscription( - &conn, - &identity_pk, - &relay_url, - scope_type.as_str(), - &scope_value, - ) + run_archive_db_task(move |conn| { + store::delete_save_subscription( + conn, + &identity_pk, + &relay_url, + scope_type.as_str(), + &scope_value, + ) + }) + .await } // ── read_archived_events ───────────────────────────────────────────────────── @@ -516,7 +518,7 @@ pub fn delete_save_subscription( /// newest-first order. Compound cursor `(before_created_at, before_id)` works /// identically to `read_archived_events`. #[tauri::command] -pub fn read_archived_observer_events_for_channel( +pub async fn read_archived_observer_events_for_channel( state: State<'_, AppState>, channel_id: String, before_created_at: Option, @@ -525,16 +527,18 @@ pub fn read_archived_observer_events_for_channel( ) -> Result, String> { let identity_pk = identity_pubkey(&state)?; let relay_url = relay_ws_url_with_override(&state); - let conn = open_db()?; - store::read_archived_observer_events_for_channel( - &conn, - &identity_pk, - &relay_url, - &channel_id, - before_created_at, - before_id.as_deref(), - limit.unwrap_or(DEFAULT_READ_LIMIT), - ) + run_archive_db_task(move |conn| { + store::read_archived_observer_events_for_channel( + conn, + &identity_pk, + &relay_url, + &channel_id, + before_created_at, + before_id.as_deref(), + limit.unwrap_or(DEFAULT_READ_LIMIT), + ) + }) + .await } // ── index_observer_channel_id ───────────────────────────────────────────────── @@ -548,24 +552,26 @@ pub fn read_archived_observer_events_for_channel( /// /// Idempotent: rows that are already indexed are left unchanged. #[tauri::command] -pub fn index_observer_channel_id( +pub async fn index_observer_channel_id( state: State<'_, AppState>, entries: Vec, ) -> Result<(), String> { let identity_pk = identity_pubkey(&state)?; let relay_url = relay_ws_url_with_override(&state); - let conn = open_db()?; - for entry in &entries { - store::upsert_observer_channel_index( - &conn, - &identity_pk, - &relay_url, - &entry.event_id, - entry.channel_id.as_deref(), - entry.created_at, - )?; - } - Ok(()) + run_archive_db_task(move |conn| { + for entry in &entries { + store::upsert_observer_channel_index( + conn, + &identity_pk, + &relay_url, + &entry.event_id, + entry.channel_id.as_deref(), + entry.created_at, + )?; + } + Ok(()) + }) + .await } /// A single (event_id, channel_id?, created_at) record used by @@ -591,21 +597,23 @@ pub struct ObserverChannelIndexEntry { /// Together these constitute the one-shot idempotent backfill required by the /// Slice 1 acceptance criteria (Thufir Pass 4). #[tauri::command] -pub fn read_unindexed_observer_rows( +pub async fn read_unindexed_observer_rows( state: State<'_, AppState>, ) -> Result, String> { let identity_pk = identity_pubkey(&state)?; let relay_url = relay_ws_url_with_override(&state); - let conn = open_db()?; - let rows = store::read_unindexed_observer_rows(&conn, &identity_pk, &relay_url)?; - Ok(rows - .into_iter() - .map(|(id, raw_json, created_at)| RawObserverRow { - id, - raw_json, - created_at, - }) - .collect()) + run_archive_db_task(move |conn| { + let rows = store::read_unindexed_observer_rows(conn, &identity_pk, &relay_url)?; + Ok(rows + .into_iter() + .map(|(id, raw_json, created_at)| RawObserverRow { + id, + raw_json, + created_at, + }) + .collect()) + }) + .await } /// Wire type returned by `read_unindexed_observer_rows`. diff --git a/desktop/src-tauri/src/commands/identity.rs b/desktop/src-tauri/src/commands/identity.rs index 33783c05a5..2840c0ade6 100644 --- a/desktop/src-tauri/src/commands/identity.rs +++ b/desktop/src-tauri/src/commands/identity.rs @@ -135,23 +135,29 @@ pub async fn sign_event( } #[tauri::command] -pub fn decrypt_observer_event( +pub async fn decrypt_observer_event( event_json: String, state: State<'_, AppState>, ) -> Result { let keys = state.signing_keys()?; - let event = Event::from_json(event_json).map_err(|error| format!("invalid event: {error}"))?; - // Defense-in-depth: verify event ID and signature before decrypting. - if !event.verify_id() { - return Err("observer event has invalid ID".into()); - } - if !event.verify_signature() { - return Err("observer event has invalid signature".into()); - } + tauri::async_runtime::spawn_blocking(move || { + let event = + Event::from_json(event_json).map_err(|error| format!("invalid event: {error}"))?; - buzz_core_pkg::observer::decrypt_observer_payload(&keys, &event) - .map_err(|error| format!("decrypt observer event failed: {error}")) + // Defense-in-depth: verify event ID and signature before decrypting. + if !event.verify_id() { + return Err("observer event has invalid ID".into()); + } + if !event.verify_signature() { + return Err("observer event has invalid signature".into()); + } + + buzz_core_pkg::observer::decrypt_observer_payload(&keys, &event) + .map_err(|error| format!("decrypt observer event failed: {error}")) + }) + .await + .map_err(|e| format!("spawn_blocking failed: {e}"))? } #[tauri::command] From 51bb97d2be658854bb8a39983af568e90591d375 Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 09:20:49 -0600 Subject: [PATCH 08/32] Run Tauri clippy in pre-push (#3555) ## Summary - run Desktop Tauri clippy from pre-push for every path that can affect the Tauri crate - reuse `just desktop-tauri-clippy`, keeping the local command identical to Desktop Core CI - leave the existing Tauri test hook unchanged ## Why PR #3553 exposed a hook gap: `cargo test` allowed an unused-import warning that CI's `clippy -D warnings` correctly rejected. Running the same recipe before push catches that class of failure locally without duplicating CI flags in Lefthook. ## Validation - `lefthook run pre-push --command desktop-tauri-clippy --force` - confirmed it invokes `cargo clippy --manifest-path desktop/src-tauri/Cargo.toml --all-targets -- -D warnings` - command passed Signed-off-by: Wes Co-authored-by: Carl --- lefthook.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lefthook.yml b/lefthook.yml index 87eaa87c21..0680dd29a4 100644 --- a/lefthook.yml +++ b/lefthook.yml @@ -61,6 +61,11 @@ pre-push: glob: ["desktop/**", "pnpm-lock.yaml"] exclude: ["desktop/src-tauri/**"] run: just desktop-test + desktop-tauri-clippy: + # Keep local lint parity with Desktop Core CI for every path that can + # affect the Tauri crate or its path dependencies. + glob: ["desktop/src-tauri/**", "crates/**", "migrations/**", "schema/**", "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "deny.toml", "scripts/run-tests.sh", "justfile"] + run: just desktop-tauri-clippy desktop-tauri-test: # ci.yml:113 — Desktop Core triggers on `rust` OR `desktop-rust`; # desktop/src-tauri path-depends on crates/buzz-core, buzz-persona, From a13085e9ac9a7c8dbd9426a6b88fc75abf62220e Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 09:44:30 -0600 Subject: [PATCH 09/32] chore(release): release Buzz Desktop version 0.5.1 (#3566) ## Buzz Desktop release v0.5.1 ### Changes since v0.5.0: - perf(desktop): move observer-feed archive and decrypt commands off main thread ([#3415](https://github.com/block/buzz/pull/3415)) ([`294c8c821`](https://github.com/block/buzz/commit/294c8c821de51442a8c384c0bdb66b1a10224ca0)) - fix(desktop): preserve shared agent fidelity ([#3553](https://github.com/block/buzz/pull/3553)) ([`f7a3988ba`](https://github.com/block/buzz/commit/f7a3988ba13b590d9a55a7e8413fc3fb5ffbef18)) - feat(agent): route Claude/GPT model families to their native gateway wire ([#3538](https://github.com/block/buzz/pull/3538)) ([`6438dedf8`](https://github.com/block/buzz/commit/6438dedf83a9dbe1853e484326911bf6c7f1618c)) - Refine community invite limits ([#3529](https://github.com/block/buzz/pull/3529)) ([`24d90d128`](https://github.com/block/buzz/commit/24d90d1280a9325c6cbcf8eea30ac54db5afd2cb)) - feat(agent): fix Anthropic prompt caching with Databricks (+ MCP proxy/TLS passthrough) ([#3463](https://github.com/block/buzz/pull/3463)) ([`c405ad1d4`](https://github.com/block/buzz/commit/c405ad1d4b1da061c11b3d26761252d41dcc62d3)) - feat: add explicit entry for claude-opus-5 in model config ([#2831](https://github.com/block/buzz/pull/2831)) ([`90e058ebf`](https://github.com/block/buzz/commit/90e058ebf68137e048a409aec6616519379ff726)) - fix(desktop): clear stale thread new-message pill ([#3411](https://github.com/block/buzz/pull/3411)) ([`55a3ed7b9`](https://github.com/block/buzz/commit/55a3ed7b9217cee5b23e0a5441947dc929b2a38c)) - fix(ci): ratchet file sizes against the base tree ([#3352](https://github.com/block/buzz/pull/3352)) ([`9227bdf58`](https://github.com/block/buzz/commit/9227bdf58ad6664ae3c1078888f2181ec19c4da4)) - feat(desktop): apply WebKit rendering workarounds at startup on Linux ([#3271](https://github.com/block/buzz/pull/3271)) ([`3ece4461d`](https://github.com/block/buzz/commit/3ece4461df8a7b9663a8e68327483b8377d4086d)) - fix(desktop): stabilize flaky DM expansion E2E ordering assertions ([#2004](https://github.com/block/buzz/pull/2004)) ([`913d564ce`](https://github.com/block/buzz/commit/913d564ce0f35924291bf3eeab6508517a6d8d1f)) - fix(desktop): paint community rail full height ([#3382](https://github.com/block/buzz/pull/3382)) ([`1d3b810ad`](https://github.com/block/buzz/commit/1d3b810ad70d6325718ed91e723f32c4a376d5e1)) - feat(desktop): add custom harness inline from agent dialogs ([#3252](https://github.com/block/buzz/pull/3252)) ([`b0503d80c`](https://github.com/block/buzz/commit/b0503d80c298b1ece3b0a43b41d316829a3379e7)) - feat(desktop): refine agent catalog sharing ([#2439](https://github.com/block/buzz/pull/2439)) ([`a35771fc4`](https://github.com/block/buzz/commit/a35771fc441cdc3c6f517f419037206783b502d2)) - fix(desktop): keep drafts out of the Inbox All view ([#3217](https://github.com/block/buzz/pull/3217)) ([`3afa129ee`](https://github.com/block/buzz/commit/3afa129ee785cc74d921d0ba969254a8255c4cc0)) - fix(desktop): restore the inbox icon in the sidebar ([#3341](https://github.com/block/buzz/pull/3341)) ([`00ede2e7a`](https://github.com/block/buzz/commit/00ede2e7aa7eb95571b7db3ebbd163adbf6cf74e)) - fix(desktop): gate codex-acp on a minimum supported version ([#3254](https://github.com/block/buzz/pull/3254)) ([`4e3998f36`](https://github.com/block/buzz/commit/4e3998f36e36d68b9a93dcbd85f0864450bb8f5f)) - feat(cli): add users set-status command for NIP-38 profile status ([#3253](https://github.com/block/buzz/pull/3253)) ([`60158fce3`](https://github.com/block/buzz/commit/60158fce3e670f11bb35d42627857ccaea50ff06)) - fix(composer): scope multiline block formatting ([#3246](https://github.com/block/buzz/pull/3246)) ([`5457c947a`](https://github.com/block/buzz/commit/5457c947a74f5ba4b979f9c6411aa7626a858387)) **To release:** merge this PR. The tag and build will happen automatically. Signed-off-by: Wes --- CHANGELOG.md | 22 ++++++++++++++++++++++ desktop/package.json | 2 +- desktop/src-tauri/Cargo.lock | 2 +- desktop/src-tauri/Cargo.toml | 2 +- desktop/src-tauri/tauri.conf.json | 2 +- 5 files changed, 26 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cfd3b16d0a..956faa1ed3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## v0.5.1 + +- perf(desktop): move observer-feed archive and decrypt commands off main thread ([#3415](https://github.com/block/buzz/pull/3415)) ([`294c8c821`](https://github.com/block/buzz/commit/294c8c821de51442a8c384c0bdb66b1a10224ca0)) +- fix(desktop): preserve shared agent fidelity ([#3553](https://github.com/block/buzz/pull/3553)) ([`f7a3988ba`](https://github.com/block/buzz/commit/f7a3988ba13b590d9a55a7e8413fc3fb5ffbef18)) +- feat(agent): route Claude/GPT model families to their native gateway wire ([#3538](https://github.com/block/buzz/pull/3538)) ([`6438dedf8`](https://github.com/block/buzz/commit/6438dedf83a9dbe1853e484326911bf6c7f1618c)) +- Refine community invite limits ([#3529](https://github.com/block/buzz/pull/3529)) ([`24d90d128`](https://github.com/block/buzz/commit/24d90d1280a9325c6cbcf8eea30ac54db5afd2cb)) +- feat(agent): fix Anthropic prompt caching with Databricks (+ MCP proxy/TLS passthrough) ([#3463](https://github.com/block/buzz/pull/3463)) ([`c405ad1d4`](https://github.com/block/buzz/commit/c405ad1d4b1da061c11b3d26761252d41dcc62d3)) +- feat: add explicit entry for claude-opus-5 in model config ([#2831](https://github.com/block/buzz/pull/2831)) ([`90e058ebf`](https://github.com/block/buzz/commit/90e058ebf68137e048a409aec6616519379ff726)) +- fix(desktop): clear stale thread new-message pill ([#3411](https://github.com/block/buzz/pull/3411)) ([`55a3ed7b9`](https://github.com/block/buzz/commit/55a3ed7b9217cee5b23e0a5441947dc929b2a38c)) +- fix(ci): ratchet file sizes against the base tree ([#3352](https://github.com/block/buzz/pull/3352)) ([`9227bdf58`](https://github.com/block/buzz/commit/9227bdf58ad6664ae3c1078888f2181ec19c4da4)) +- feat(desktop): apply WebKit rendering workarounds at startup on Linux ([#3271](https://github.com/block/buzz/pull/3271)) ([`3ece4461d`](https://github.com/block/buzz/commit/3ece4461df8a7b9663a8e68327483b8377d4086d)) +- fix(desktop): stabilize flaky DM expansion E2E ordering assertions ([#2004](https://github.com/block/buzz/pull/2004)) ([`913d564ce`](https://github.com/block/buzz/commit/913d564ce0f35924291bf3eeab6508517a6d8d1f)) +- fix(desktop): paint community rail full height ([#3382](https://github.com/block/buzz/pull/3382)) ([`1d3b810ad`](https://github.com/block/buzz/commit/1d3b810ad70d6325718ed91e723f32c4a376d5e1)) +- feat(desktop): add custom harness inline from agent dialogs ([#3252](https://github.com/block/buzz/pull/3252)) ([`b0503d80c`](https://github.com/block/buzz/commit/b0503d80c298b1ece3b0a43b41d316829a3379e7)) +- feat(desktop): refine agent catalog sharing ([#2439](https://github.com/block/buzz/pull/2439)) ([`a35771fc4`](https://github.com/block/buzz/commit/a35771fc441cdc3c6f517f419037206783b502d2)) +- fix(desktop): keep drafts out of the Inbox All view ([#3217](https://github.com/block/buzz/pull/3217)) ([`3afa129ee`](https://github.com/block/buzz/commit/3afa129ee785cc74d921d0ba969254a8255c4cc0)) +- fix(desktop): restore the inbox icon in the sidebar ([#3341](https://github.com/block/buzz/pull/3341)) ([`00ede2e7a`](https://github.com/block/buzz/commit/00ede2e7aa7eb95571b7db3ebbd163adbf6cf74e)) +- fix(desktop): gate codex-acp on a minimum supported version ([#3254](https://github.com/block/buzz/pull/3254)) ([`4e3998f36`](https://github.com/block/buzz/commit/4e3998f36e36d68b9a93dcbd85f0864450bb8f5f)) +- feat(cli): add users set-status command for NIP-38 profile status ([#3253](https://github.com/block/buzz/pull/3253)) ([`60158fce3`](https://github.com/block/buzz/commit/60158fce3e670f11bb35d42627857ccaea50ff06)) +- fix(composer): scope multiline block formatting ([#3246](https://github.com/block/buzz/pull/3246)) ([`5457c947a`](https://github.com/block/buzz/commit/5457c947a74f5ba4b979f9c6411aa7626a858387)) + + ## v0.5.0 - feat(invites): add use-limited invite links ([#3141](https://github.com/block/buzz/pull/3141)) ([`d500c2d5c`](https://github.com/block/buzz/commit/d500c2d5cf5d9aabe0ca4ebebfcafdbe5f5b7fd3)) diff --git a/desktop/package.json b/desktop/package.json index adac095a47..7943b949b9 100644 --- a/desktop/package.json +++ b/desktop/package.json @@ -1,7 +1,7 @@ { "name": "buzz", "private": true, - "version": "0.5.0", + "version": "0.5.1", "type": "module", "scripts": { "dev": "vite", diff --git a/desktop/src-tauri/Cargo.lock b/desktop/src-tauri/Cargo.lock index 66553ef595..d4f7a4a2d4 100644 --- a/desktop/src-tauri/Cargo.lock +++ b/desktop/src-tauri/Cargo.lock @@ -1010,7 +1010,7 @@ dependencies = [ [[package]] name = "buzz-desktop" -version = "0.5.0" +version = "0.5.1" dependencies = [ "anyhow", "arboard", diff --git a/desktop/src-tauri/Cargo.toml b/desktop/src-tauri/Cargo.toml index 324218a49d..8bb643fea3 100644 --- a/desktop/src-tauri/Cargo.toml +++ b/desktop/src-tauri/Cargo.toml @@ -2,7 +2,7 @@ [package] name = "buzz-desktop" -version = "0.5.0" +version = "0.5.1" description = "Buzz desktop app" authors = ["you"] edition = "2021" diff --git a/desktop/src-tauri/tauri.conf.json b/desktop/src-tauri/tauri.conf.json index 7a480c4c18..85ad5c0b2d 100644 --- a/desktop/src-tauri/tauri.conf.json +++ b/desktop/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "Buzz", - "version": "0.5.0", + "version": "0.5.1", "identifier": "xyz.block.buzz.app", "build": { "beforeDevCommand": { From 9752b816a95e88751b9c7e4d3343fe3210de5d43 Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 09:52:50 -0600 Subject: [PATCH 10/32] Serialize Tauri pre-push checks (#3567) ## Summary - combine Desktop Tauri clippy and tests into one pre-push command - run clippy first, then tests - keep unrelated pre-push commands parallel ## Why PR #3555 added clippy as a separate command while the pre-push group uses `parallel: true`. That can start clippy and tests simultaneously against the same Cargo target directory, leaving one command waiting on Cargo's build lock and making pushes appear stalled. Serializing only these two Cargo-heavy checks avoids lock contention while retaining the CI-equivalent clippy command and existing test coverage. ## Validation - `lefthook validate` - forced `desktop-tauri-checks` through Lefthook with an instrumented `just`; observed `desktop-tauri-clippy` followed by `desktop-tauri-test` Signed-off-by: Wes Co-authored-by: Carl --- lefthook.yml | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/lefthook.yml b/lefthook.yml index 0680dd29a4..75d205722f 100644 --- a/lefthook.yml +++ b/lefthook.yml @@ -61,17 +61,12 @@ pre-push: glob: ["desktop/**", "pnpm-lock.yaml"] exclude: ["desktop/src-tauri/**"] run: just desktop-test - desktop-tauri-clippy: + desktop-tauri-checks: # Keep local lint parity with Desktop Core CI for every path that can - # affect the Tauri crate or its path dependencies. + # affect the Tauri crate or its path dependencies. Run clippy and tests + # serially so parallel pre-push hooks do not contend for Cargo's lock. glob: ["desktop/src-tauri/**", "crates/**", "migrations/**", "schema/**", "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "deny.toml", "scripts/run-tests.sh", "justfile"] - run: just desktop-tauri-clippy - desktop-tauri-test: - # ci.yml:113 — Desktop Core triggers on `rust` OR `desktop-rust`; - # desktop/src-tauri path-depends on crates/buzz-core, buzz-persona, - # buzz-sdk, buzz-agent (desktop/src-tauri/Cargo.toml:88-91). - glob: ["desktop/src-tauri/**", "crates/**", "migrations/**", "schema/**", "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "deny.toml", "scripts/run-tests.sh", "justfile"] - run: just desktop-tauri-test + run: just desktop-tauri-clippy && just desktop-tauri-test mobile-test: glob: ["mobile/**"] run: just mobile-test From 9beb3b8c6ebc3e2fb0c17f4ca4d522731b81434c Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 12:23:02 -0400 Subject: [PATCH 11/32] fix(cli): mask credential env values in --help output (#3570) clap renders live env var values in help text by default. Three args carrying credentials were exposed this way: - `BUZZ_PRIVATE_KEY` in `buzz-cli` (`crates/buzz-cli/src/lib.rs`) - `BUZZ_AUTH_TAG` in `buzz-cli` - `BUZZ_PRIVATE_KEY` in `buzz-acp` (`crates/buzz-acp/src/config.rs`) Add `hide_env_values = true` to each. Env var names remain visible for discoverability; only their runtime values are withheld from `--help` output. Also adds a regression guard in each crate's test module that walks the clap command tree (recursing into subcommands for `buzz-cli`) and asserts every arg whose env var name contains `KEY`, `SECRET`, `TOKEN`, `PASSWORD`, `CRED`, or `AUTH` has `hide_env_values` set. This prevents future credential-bearing args from being added without the masking in place. Signed-off-by: Will Pfleger Co-authored-by: Duncan --- crates/buzz-acp/src/config.rs | 32 +++++++++++++++++++++++- crates/buzz-cli/src/lib.rs | 46 +++++++++++++++++++++++++++++++++-- 2 files changed, 75 insertions(+), 3 deletions(-) diff --git a/crates/buzz-acp/src/config.rs b/crates/buzz-acp/src/config.rs index 19304bf186..dab61be30a 100644 --- a/crates/buzz-acp/src/config.rs +++ b/crates/buzz-acp/src/config.rs @@ -240,7 +240,7 @@ pub struct CliArgs { #[arg(long, env = "BUZZ_RELAY_URL", default_value = "ws://localhost:3000")] pub relay_url: String, - #[arg(long, env = "BUZZ_PRIVATE_KEY")] + #[arg(long, env = "BUZZ_PRIVATE_KEY", hide_env_values = true)] pub private_key: String, /// Agent owner pubkey (64-char hex). Used for --respond-to=owner-only gate. @@ -2898,4 +2898,34 @@ channels = "ALL" let agent = "a".repeat(SESSION_TITLE_MAX_CHARS); assert_eq!(compose_session_title(&agent, Some("buzz-dev")), agent); } + + /// Every arg whose env var name contains KEY/SECRET/TOKEN/PASSWORD/CRED/AUTH + /// must set `hide_env_values = true` to prevent credential leakage in --help. + #[test] + fn secret_env_args_hide_their_values_in_help() { + use clap::CommandFactory; + + const SECRET_PATTERNS: &[&str] = &["KEY", "SECRET", "TOKEN", "PASSWORD", "CRED", "AUTH"]; + + let cmd = CliArgs::command(); + let violations: Vec = cmd + .get_arguments() + .filter_map(|arg| { + let env_key = arg.get_env()?; + let env_name = env_key.to_string_lossy().to_uppercase(); + let is_secret = SECRET_PATTERNS.iter().any(|pat| env_name.contains(pat)); + if is_secret && !arg.is_hide_env_values_set() { + Some(env_name) + } else { + None + } + }) + .collect(); + + assert!( + violations.is_empty(), + "Found secret-bearing env args without hide_env_values=true. \ + Add `hide_env_values = true` to each: {violations:?}" + ); + } } diff --git a/crates/buzz-cli/src/lib.rs b/crates/buzz-cli/src/lib.rs index 0b46734584..7465625804 100644 --- a/crates/buzz-cli/src/lib.rs +++ b/crates/buzz-cli/src/lib.rs @@ -82,11 +82,11 @@ struct Cli { relay: String, /// Nostr private key (hex or nsec). This is the CLI's identity. - #[arg(long, env = "BUZZ_PRIVATE_KEY")] + #[arg(long, env = "BUZZ_PRIVATE_KEY", hide_env_values = true)] private_key: Option, /// NIP-OA auth tag JSON (owner attestation). Injected into every signed event. - #[arg(long, env = "BUZZ_AUTH_TAG")] + #[arg(long, env = "BUZZ_AUTH_TAG", hide_env_values = true)] auth_tag: Option, /// Output format: 'json' (default, full fields) or 'compact' (reduced fields). @@ -2075,4 +2075,46 @@ mod tests { ); } } + + /// Collect all args (recursing into subcommands) whose env var name looks + /// like a credential but does NOT have `hide_env_values` set. + fn collect_unhidden_secret_args(cmd: &clap::Command) -> Vec<(String, String)> { + const SECRET_PATTERNS: &[&str] = &["KEY", "SECRET", "TOKEN", "PASSWORD", "CRED", "AUTH"]; + + let mut violations: Vec<(String, String)> = Vec::new(); + + for arg in cmd.get_arguments() { + if let Some(env_key) = arg.get_env() { + let env_name = env_key.to_string_lossy().to_uppercase(); + let is_secret = SECRET_PATTERNS.iter().any(|pat| env_name.contains(pat)); + if is_secret && !arg.is_hide_env_values_set() { + violations.push((cmd.get_name().to_string(), env_name)); + } + } + } + + for sub in cmd.get_subcommands() { + violations.extend(collect_unhidden_secret_args(sub)); + } + + violations + } + + /// Every arg whose env var name contains KEY/SECRET/TOKEN/PASSWORD/CRED/AUTH + /// must set `hide_env_values = true` to prevent credential leakage in --help. + #[test] + fn secret_env_args_hide_their_values_in_help() { + let cmd = Cli::command(); + let violations = collect_unhidden_secret_args(&cmd); + assert!( + violations.is_empty(), + "Found secret-bearing env args without hide_env_values=true. \ + Add `hide_env_values = true` to each:\n{}", + violations + .iter() + .map(|(cmd, env)| format!(" command={cmd:?} env={env:?}")) + .collect::>() + .join("\n") + ); + } } From 4a1ebf25c782fc6a68f0a69e6f866f793a259a1f Mon Sep 17 00:00:00 2001 From: Atish Patel Date: Wed, 29 Jul 2026 11:49:51 -0500 Subject: [PATCH 12/32] feat(agent): make Gemini and MLflow-route models usable through databricks_v2 (#3569) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Makes Gemini — and every other non-Claude, non-GPT-5 model on the Databricks MLflow route (`databricks_v2`) — usable in an agent loop. These are the non-`benchmarks/` changes from `benchmark/harness-accounting-and-solo`, lifted onto a clean base off `main` so they can land independently while the harness work continues. Two defects made these models unusable, one fatal and one silent. Both live only in `openai_body` / `parse_openai`, which is the least-exercised of the three `databricks_v2` sub-routes — the `luna`/`sol` conditions run the Responses route and the `opus` conditions run the Anthropic route, so **this change is inert for every model already in use** and only lights up the MLflow path. ## Why a third route at all `databricks_v2_route_for_model` buckets by model family: `claude*` → Anthropic Messages, `gpt-5`/code-names → OpenAI Responses, **everything else → MLflow chat-completions**. Gemini, Qwen, gpt-oss, and friends all fall through to that third pair — and both bugs below live only there. ## D1 — dropped thought signatures (fatal) Gemini returns a `thoughtSignature` on every tool call and **requires it echoed back**. `openai_body` reserialized each call as `{id, type, function}` only, dropping the field, so the next request 400'd: ``` HTTP 400 Function call is missing a thought_signature in functionCall parts. ``` For a coding agent this fires on the **first** tool call, so the model never completes a single turn. **Position is load-bearing.** A four-shape replay probe against the live gateway established that the signature must sit as a *sibling* of `function` — nesting it inside `function{}` fails with the *same* 400 as omitting it. A fix that "preserves the field" without preserving its position passes a unit test and still 400s. The fix: `ToolCall` gains `provider_extra: Map`. `parse_openai` captures every top-level wire key except the three we model (`id`, `type`, `function`); `openai_body` re-emits them beside `function`. Keeping *whatever we did not model*, rather than naming `thoughtSignature`, means the next provider with an opaque per-call token needs no change here. The Responses and Anthropic replay shapes are fully modelled, so they pass `Default::default()` and stay **byte-identical** to before. ### D1b — duplicate tool-call ids (same root cause) Gemini returns the **function name** as the id, so two parallel calls to one function arrive sharing an id — and that id is what pairs a `role:"tool"` result back to its call, making two results indistinguishable. `dedupe_provider_ids` suffixes collisions (`get_weather`, `get_weather-2`). Safe because both halves of the pairing (the assistant `tool_calls[].id` and the result's `tool_call_id`) are re-emitted from this same value; the provider never sees its original id again. ## D2 — block-array content discarded (silent, worse than a crash) `parse_openai` read `content` with `as_str()`, which returns `""` for anything that isn't a JSON string. Gemini (and Qwen35, gpt-oss) send an array of typed blocks: ```json "content": [ {"type": "reasoning", "summary": [{"type": "summary_text", "text": "…"}]}, {"type": "text", "text": "391"} ] ``` So the model answered and the answer was thrown away — no error, no warning, just a turn that looked like the model had said nothing. On a benchmark this reads as "Gemini is bad at the task" rather than "buzz dropped the reply." `openai_content_parts` now accepts either shape — string as before, or a block array where `text` blocks concatenate into text and `reasoning` blocks into reasoning (Gemini nests the prose one level down under `summary`). Message-level `reasoning_content` / `reasoning` still win when present, so DeepSeek and vLLM-style hosts are unchanged; block reasoning is the last fallback. ## Also: a turn-start log line (`buzz-acp` `pool.rs`) Small, independent observability change that also rides in the non-benchmark delta: `run_prompt_task` now emits a `pool::prompt` "turn starting" line, labelled by the same `prompt_label` helper as `log_stop_reason`, so a log reads as start/stop pairs. An unpaired start is the only durable evidence that a turn was entered and never returned — without it, a stalled agent and an agent nobody woke leave identical (zero-completion) logs. ## Interaction with #3538 #3538 (already merged) rewrote `databricks_v2_route_for_model` to route by boundary-aware model-family segments. That change and this one touch **different functions** in `llm.rs` — routing vs. body/parse — and compose cleanly; the family routing decides *which* pair runs, and this fixes the MLflow pair it can now select. ## Testing - `cargo fmt --all -- --check`, `cargo clippy -p buzz-agent -p buzz-acp --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent -p buzz-acp` — all green (304 + 632 lib tests plus integration suites, 0 failures). Five new tests cover: block-array text extraction, plain-string regression, passthrough capture (and non-duplication of the modelled keys), replay position (`thoughtSignature` beside `function`, not inside it), and id de-duplication. - Wire evidence: the four-shape replay table and the reasoning-effort probe were run against `block-lakehouse-staging` (recorded in the design doc). ## Relationship to the benchmark branch The full design write-up (four-shape replay table, position-matters analysis, effort verification, and open pricing item) lives in `docs/08-gemini-provider-fixes.md` on `benchmark/harness-accounting-and-solo`. The benchmark manifests and endpoint-config entries that exercise these models are separable and stay on that branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Atish Patel Co-authored-by: Claude Opus 4.8 (1M context) --- crates/buzz-acp/src/pool.rs | 27 +++- crates/buzz-agent/src/agent.rs | 3 + crates/buzz-agent/src/llm.rs | 227 +++++++++++++++++++++++++++++++-- crates/buzz-agent/src/types.rs | 53 +++++++- 4 files changed, 294 insertions(+), 16 deletions(-) diff --git a/crates/buzz-acp/src/pool.rs b/crates/buzz-acp/src/pool.rs index b1fd68d044..038f8a714c 100644 --- a/crates/buzz-acp/src/pool.rs +++ b/crates/buzz-acp/src/pool.rs @@ -1897,6 +1897,18 @@ pub async fn run_prompt_task( None => prompt_sections.iter().map(String::as_str).collect(), }; + // Turn start, labelled exactly as `log_stop_reason` labels the end, so a + // log reads as start/stop pairs. Purely observational: an unpaired start is + // the only durable evidence that a turn was entered and never returned, and + // without it a stalled agent and an agent nobody woke leave identical logs — + // zero completions either way, so anything reading them afterwards has to + // guess which happened. + tracing::info!( + target: "pool::prompt", + "turn starting for {}", + prompt_label(&source) + ); + // When control_rx is Some (channel tasks), wrap the prompt in select! so // the main loop can cancel, interrupt, or rotate it. Heartbeats // (control_rx=None) take the simple await path — they are not controllable. @@ -3131,12 +3143,19 @@ fn classify_control_cancel_failure( } } -/// Log a stop reason at the appropriate tracing level. -fn log_stop_reason(source: &PromptSource, stop_reason: &StopReason) { - let label = match source { +/// How a turn's source is named in the `pool::prompt` log lines. +/// +/// Shared by the turn-start and turn-stop lines so a log can be read as pairs. +fn prompt_label(source: &PromptSource) -> String { + match source { PromptSource::Channel(cid) => format!("channel {cid}"), PromptSource::Heartbeat => "heartbeat".to_string(), - }; + } +} + +/// Log a stop reason at the appropriate tracing level. +fn log_stop_reason(source: &PromptSource, stop_reason: &StopReason) { + let label = prompt_label(source); match stop_reason { StopReason::EndTurn => { tracing::info!(target: "pool::prompt", "turn complete for {label}: end_turn"); diff --git a/crates/buzz-agent/src/agent.rs b/crates/buzz-agent/src/agent.rs index cbf27357f7..ed04daca2c 100644 --- a/crates/buzz-agent/src/agent.rs +++ b/crates/buzz-agent/src/agent.rs @@ -696,6 +696,9 @@ pub(crate) fn push_hook_outputs_as_tool_results( provider_id: provider_id.clone(), name: tool_name, arguments: serde_json::json!({}), + // Synthesised locally, so there is no provider wire form to + // preserve. + provider_extra: Default::default(), }], }); history.push(HistoryItem::ToolResult(ToolResult { diff --git a/crates/buzz-agent/src/llm.rs b/crates/buzz-agent/src/llm.rs index f69a963533..22d3f8b73e 100644 --- a/crates/buzz-agent/src/llm.rs +++ b/crates/buzz-agent/src/llm.rs @@ -3,7 +3,7 @@ use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; use reqwest::Client; -use serde_json::{json, Value}; +use serde_json::{json, Map, Value}; use tokio::sync::Mutex; use tokio::time::Instant; @@ -848,11 +848,24 @@ fn openai_body( let calls: Vec = tool_calls .iter() .map(|c| { - json!({ - "id": c.provider_id, "type": "function", - "function": { "name": c.name, - "arguments": serde_json::to_string(&c.arguments) - .unwrap_or_else(|_| "{}".into()) } }) + let mut call = serde_json::Map::new(); + call.insert("id".into(), json!(c.provider_id)); + call.insert("type".into(), json!("function")); + // Provider-owned fields go back beside `function`, + // which is where the provider put them. Position is + // load-bearing, not cosmetic: Gemini rejects a + // `thoughtSignature` nested inside `function{}` with + // the same 400 it gives for one that is missing. + for (k, v) in &c.provider_extra { + call.insert(k.clone(), v.clone()); + } + call.insert( + "function".into(), + json!({ "name": c.name, + "arguments": serde_json::to_string(&c.arguments) + .unwrap_or_else(|_| "{}".into()) }), + ); + Value::Object(call) }) .collect(); msg.insert("tool_calls".into(), Value::Array(calls)); @@ -1120,10 +1133,15 @@ fn parse_responses(v: Value) -> Result { let args: Value = serde_json::from_str(raw).map_err(|e| { AgentError::Llm(format!("function_call.arguments not valid JSON: {e}")) })?; + // No passthrough on this route: `responses_body` replays a + // function call as `{call_id, name, arguments}` and the Responses + // API asks for nothing else, so an empty map keeps the request + // byte-identical to before. tool_calls.push(make_tool_call( str_field(item, "call_id"), str_field(item, "name"), args, + Default::default(), )?); } Some("reasoning") => { @@ -1301,6 +1319,76 @@ fn str_field(v: &Value, key: &str) -> String { v.get(key).and_then(Value::as_str).unwrap_or("").to_owned() } +/// Append `part` to `buf` on its own line, ignoring empties. +fn push_part(buf: &mut String, part: &str) { + if part.is_empty() { + return; + } + if !buf.is_empty() { + buf.push('\n'); + } + buf.push_str(part); +} + +/// Split an OpenAI-shaped `message.content` into `(text, reasoning)`. +/// +/// Standard OpenAI sends a string. Several models on the Databricks MLflow route +/// — Gemini, Qwen35, gpt-oss — send an array of typed blocks instead, and +/// `as_str()` yields nothing for an array, so their entire answer was being +/// discarded: no error, no warning, just a turn that looked like the model had +/// said nothing. `parse_anthropic` already walks a block array; this gives +/// `parse_openai` the same tolerance. +fn openai_content_parts(content: Option<&Value>) -> (String, String) { + let mut text = String::new(); + let mut reasoning = String::new(); + match content { + Some(Value::String(s)) => text.push_str(s), + Some(Value::Array(blocks)) => { + for b in blocks { + match b.get("type").and_then(Value::as_str) { + Some("text") => push_part(&mut text, &str_field(b, "text")), + Some("reasoning") => match b.get("summary").and_then(Value::as_array) { + // Gemini nests the prose one level down under `summary`. + Some(summary) => { + for s in summary { + push_part(&mut reasoning, &str_field(s, "text")); + } + } + None => push_part(&mut reasoning, &str_field(b, "text")), + }, + // An untyped block carrying text is still the model talking; + // treating it as text loses nothing and keeps one more + // provider out of the silent-empty-answer failure mode. + _ => push_part(&mut text, &str_field(b, "text")), + } + } + } + _ => {} + } + (text, reasoning) +} + +/// Make `provider_id` unique across one assistant turn's tool calls. +/// +/// Gemini returns the function name as the id, so two parallel calls to the same +/// function arrive sharing one id — and that id is what pairs a `role:"tool"` +/// result back to its call, leaving two results indistinguishable. Rewriting is +/// safe because both halves of that pairing are re-emitted from this same value; +/// the provider never sees its original id again. +fn dedupe_provider_ids(calls: &mut [ToolCall]) { + let mut seen: BTreeSet = BTreeSet::new(); + for c in calls.iter_mut() { + if seen.contains(&c.provider_id) { + let mut n = 2; + while seen.contains(&format!("{}-{n}", c.provider_id)) { + n += 1; + } + c.provider_id = format!("{}-{n}", c.provider_id); + } + seen.insert(c.provider_id.clone()); + } +} + fn parse_anthropic(v: Value) -> Result { let stop = map_stop(v.get("stop_reason").and_then(Value::as_str)); let mut tool_calls = Vec::new(); @@ -1323,10 +1411,12 @@ fn parse_anthropic(v: Value) -> Result { reasoning.push_str(t); } } + // Anthropic's replay shape is fully modelled, so nothing to keep. Some("tool_use") => tool_calls.push(make_tool_call( str_field(b, "id"), str_field(b, "name"), b.get("input").cloned().unwrap_or(Value::Null), + Default::default(), )?), _ => {} } @@ -1358,17 +1448,23 @@ fn parse_openai(v: Value) -> Result { let msg = choice .get("message") .ok_or_else(|| AgentError::Llm("missing message".into()))?; - let text = str_field(msg, "content"); + let (text, block_reasoning) = openai_content_parts(msg.get("content")); // DeepSeek and vLLM-style OpenAI-compat hosts expose reasoning tokens on the // message object. Prefer `reasoning_content` (DeepSeek's field name); fall - // back to `reasoning` (some other providers). Both are absent for standard - // OpenAI responses, which leaves this empty without any special-casing. + // back to `reasoning` (some other providers), and last to reasoning blocks + // found inside `content`. All three are absent for standard OpenAI + // responses, which leaves this empty without any special-casing. let reasoning = { let rc = str_field(msg, "reasoning_content"); - if rc.is_empty() { + let rc = if rc.is_empty() { str_field(msg, "reasoning") } else { rc + }; + if rc.is_empty() { + block_reasoning + } else { + rc } }; let mut tool_calls = Vec::new(); @@ -1380,13 +1476,25 @@ fn parse_openai(v: Value) -> Result { let raw = f.get("arguments").and_then(Value::as_str).unwrap_or("{}"); let args: Value = serde_json::from_str(raw) .map_err(|e| AgentError::Llm(format!("tool_call.arguments not valid JSON: {e}")))?; + // Everything on the wire object we do not model, kept for replay. + let extra = tc + .as_object() + .map(|o| { + o.iter() + .filter(|(k, _)| !matches!(k.as_str(), "id" | "type" | "function")) + .map(|(k, v)| (k.clone(), v.clone())) + .collect() + }) + .unwrap_or_default(); tool_calls.push(make_tool_call( str_field(tc, "id"), str_field(f, "name"), args, + extra, )?); } } + dedupe_provider_ids(&mut tool_calls); let input_tokens = openai_chat_input_tokens(&v); let output_tokens = sum_usage(&v, &["completion_tokens"]); let cached_input_tokens = openai_chat_cached_tokens(&v); @@ -1401,7 +1509,12 @@ fn parse_openai(v: Value) -> Result { }) } -fn make_tool_call(id: String, name: String, args: Value) -> Result { +fn make_tool_call( + id: String, + name: String, + args: Value, + provider_extra: Map, +) -> Result { if id.is_empty() || name.is_empty() { return Err(AgentError::Llm("tool_call missing id or name".into())); } @@ -1418,6 +1531,7 @@ fn make_tool_call(id: String, name: String, args: Value) -> Result Value { + json!({"choices": [{"finish_reason": "tool_calls", "message": { + "role": "assistant", + "content": [ + {"type": "reasoning", "summary": [{"type": "summary_text", "text": "weighing it"}]}, + {"type": "text", "text": "391"} + ], + "tool_calls": [{ + "id": "get_weather", "type": "function", "thoughtSignature": "SIG-A", + "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"} + }] + }}]}) + } + + #[test] + fn parse_openai_reads_text_out_of_a_block_array() { + // Before this, `as_str()` on the array yielded "" and the model's answer + // was discarded with no error at all. + let r = parse_openai(gemini_choice()).unwrap(); + assert_eq!(r.text, "391"); + assert_eq!(r.reasoning, "weighing it"); + } + + #[test] + fn parse_openai_still_reads_a_plain_string_content() { + let v = json!({"choices": [{"finish_reason": "stop", "message": { + "role": "assistant", "content": "plain"}}]}); + let r = parse_openai(v).unwrap(); + assert_eq!(r.text, "plain"); + assert_eq!(r.reasoning, ""); + } + + #[test] + fn parse_openai_keeps_unmodelled_tool_call_fields() { + let r = parse_openai(gemini_choice()).unwrap(); + let extra = &r.tool_calls[0].provider_extra; + assert_eq!(extra.get("thoughtSignature"), Some(&json!("SIG-A"))); + // `id`/`type`/`function` are modelled, so they must not be duplicated + // into the passthrough — they would be re-emitted twice. + assert!(!extra.contains_key("id")); + assert!(!extra.contains_key("type")); + assert!(!extra.contains_key("function")); + } + + #[test] + fn openai_body_replays_the_signature_beside_function_not_inside_it() { + // Position is what the gateway checks: nested inside `function{}` it is + // rejected with the same 400 as a missing signature. + let r = parse_openai(gemini_choice()).unwrap(); + let history = vec![HistoryItem::Assistant { + text: r.text.clone(), + tool_calls: r.tool_calls.clone(), + }]; + let body = openai_body( + &cfg(Provider::DatabricksV2), + "sys", + &history, + &[], + "databricks-gemini-3-6-flash", + None, + ); + let call = &body["messages"][1]["tool_calls"][0]; + assert_eq!(call["thoughtSignature"], json!("SIG-A")); + assert!(call["function"].get("thoughtSignature").is_none()); + assert_eq!(call["function"]["name"], json!("get_weather")); + } + + #[test] + fn parse_openai_makes_duplicate_tool_call_ids_unique() { + // Gemini returns the function name as the id, so parallel calls to one + // function collide and their results become indistinguishable. + let v = json!({"choices": [{"finish_reason": "tool_calls", "message": { + "role": "assistant", "content": "", + "tool_calls": [ + {"id": "get_weather", "type": "function", + "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}}, + {"id": "get_weather", "type": "function", + "function": {"name": "get_weather", "arguments": "{\"city\":\"Rome\"}"}} + ]}}]}); + let r = parse_openai(v).unwrap(); + assert_eq!(r.tool_calls[0].provider_id, "get_weather"); + assert_eq!(r.tool_calls[1].provider_id, "get_weather-2"); + } + #[test] fn parse_openai_uses_prompt_tokens() { let v = serde_json::json!({ diff --git a/crates/buzz-agent/src/types.rs b/crates/buzz-agent/src/types.rs index 1b5f30b1ce..a3d48a7cf1 100644 --- a/crates/buzz-agent/src/types.rs +++ b/crates/buzz-agent/src/types.rs @@ -1,5 +1,5 @@ use serde::Deserialize; -use serde_json::Value; +use serde_json::{Map, Value}; /// Byte-equivalent charged to the handoff/context-pressure gate for a single /// image tool result. The gate maps bytes to tokens at 1 byte/token (see @@ -93,6 +93,13 @@ impl HistoryItem { + serde_json::to_vec(&c.arguments) .map(|b| b.len()) .unwrap_or(0) + // `provider_extra` (e.g. a Gemini + // `thoughtSignature`) is re-serialized into + // every replayed call, so it counts toward the + // request body and the context-pressure gate. + + serde_json::to_vec(&c.provider_extra) + .map(|b| b.len()) + .unwrap_or(0) }) .sum::() } @@ -108,6 +115,17 @@ pub struct ToolCall { pub provider_id: String, pub name: String, pub arguments: Value, + /// Fields the provider put on the tool call that we do not model, kept so + /// the assistant turn can be replayed the way it arrived. + /// + /// Gemini on the Databricks MLflow route returns a `thoughtSignature` per + /// call and *requires* it echoed back: replaying without it fails the whole + /// request with `Function call is missing a thought_signature in functionCall + /// parts`. For an agent loop that lands on the very first tool call, so the + /// model is unusable without this. Carrying whatever we did not model, + /// rather than naming that one field, means the next provider with an opaque + /// per-call token needs no change here. + pub provider_extra: Map, } #[derive(Debug, Clone)] @@ -353,6 +371,39 @@ mod tests { assert!(item.estimated_bytes() >= 3_118_884); } + #[test] + fn assistant_size_counts_provider_extra() { + // A Gemini `thoughtSignature` rides the wire on every replayed call, so + // both size measures must see it — otherwise `truncate_history` and the + // handoff gate under-count and let the real request exceed the budget. + let mut extra = Map::new(); + extra.insert("thoughtSignature".into(), Value::String("S".repeat(500))); + let with_extra = HistoryItem::Assistant { + text: String::new(), + tool_calls: vec![ToolCall { + provider_id: "id".into(), + name: "t".into(), + arguments: Value::Null, + provider_extra: extra, + }], + }; + let without_extra = HistoryItem::Assistant { + text: String::new(), + tool_calls: vec![ToolCall { + provider_id: "id".into(), + name: "t".into(), + arguments: Value::Null, + provider_extra: Map::new(), + }], + }; + assert!(with_extra.estimated_bytes() > without_extra.estimated_bytes() + 500); + assert_eq!( + with_extra.estimated_bytes(), + with_extra.context_pressure_bytes(), + "provider_extra is text, so both measures must agree" + ); + } + #[test] fn text_content_size_is_identical_for_both_measures() { // Only images diverge; text must size the same under both paths. From ddd468723ac1f0663abd19d23ca3484b2dd40bf5 Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 13:08:32 -0400 Subject: [PATCH 13/32] revert(acp): remove dead GOOSE_ACP_SCHEDULER_DISABLED env injection (#3576) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary [block/buzz#3144](https://github.com/block/buzz/pull/3144) injected `GOOSE_ACP_SCHEDULER_DISABLED=true` into every `AcpClient::spawn` call as a forward-compatible no-op, intended to suppress the cron scheduler in goose ACP children once the matching reader landed in goose. That reader only ever existed in [aaif-goose/goose#10738](https://github.com/aaif-goose/goose/pull/10738), which was closed unmerged. [goose#10781](https://github.com/aaif-goose/goose/pull/10781) (Lifei Zhou, merged 2026-07-29) disables the ACP scheduler by default at the source: `goose acp` now requires `--enable-scheduler` to start a scheduler. Buzz-spawned children therefore get no scheduler with zero configuration — making the `GOOSE_ACP_SCHEDULER_DISABLED` injection permanently dead code. ## What changes Removes from `crates/buzz-acp/src/acp.rs`: - `GOOSE_SCHEDULER_DISABLED_ENV` constant - `cmd.env(GOOSE_SCHEDULER_DISABLED_ENV, "true")` injection in `AcpClient::spawn` - `spawn_injects_scheduler_disabled_env_by_default` test - `spawn_scheduler_disabled_env_overrides_conflicting_extra_env` test - `spawn_and_read_child_env` helper (unreferenced once the two tests above are gone) No other files are affected. ## Why now Leaving dead code that references an env var no reader will ever consume misleads future maintainers about the actual scheduler-isolation mechanism. The isolation is now an upstream default, not a Buzz injection. Reverts: [block/buzz#3144](https://github.com/block/buzz/pull/3144) Related: [aaif-goose/goose#10781](https://github.com/aaif-goose/goose/pull/10781) Signed-off-by: Will Pfleger --- crates/buzz-acp/src/acp.rs | 85 -------------------------------------- 1 file changed, 85 deletions(-) diff --git a/crates/buzz-acp/src/acp.rs b/crates/buzz-acp/src/acp.rs index 3514580519..8a698954a0 100644 --- a/crates/buzz-acp/src/acp.rs +++ b/crates/buzz-acp/src/acp.rs @@ -20,10 +20,6 @@ use crate::usage::{TurnUsage, UsageTracker}; /// Lines exceeding this limit are rejected to prevent OOM from rogue agents. const MAX_LINE_SIZE: usize = 10_000_000; // 10 MB -/// Env var that tells a goose ACP child not to start its cron scheduler. -/// Injected unconditionally by [`AcpClient::spawn`]; see the call site for why. -pub(crate) const GOOSE_SCHEDULER_DISABLED_ENV: &str = "GOOSE_ACP_SCHEDULER_DISABLED"; - /// An MCP server configuration passed to `session/new`. /// /// Corresponds to the `McpServerStdio` variant in the ACP schema. @@ -517,16 +513,6 @@ impl AcpClient { cmd.env("CODEX_CONFIG", merged); } - // Buzz-managed agents must never execute the operator's personal cron - // schedule. A goose ACP child starts a scheduler over the shared - // `schedule.json`, so a pool of N children fires every scheduled job N - // times — under the wrong identity and racing standalone goose. - // - // Set last, and with no operator-wins escape hatch, so it beats both a - // conflicting persona `extra_env` entry and any inherited parent value. - // Agent builds that don't recognize the variable ignore it. - cmd.env(GOOSE_SCHEDULER_DISABLED_ENV, "true"); - // Spawn the agent in its own process group so SIGKILL doesn't propagate // to the harness's own process group on Unix. // tokio::process::Command::process_group is a stable tokio API (no extra imports needed). @@ -2866,46 +2852,6 @@ mod tests { .expect("failed to spawn test script") } - /// Spawn a script that echoes the named env vars as the child observes - /// them, one per line. `` means the child did not receive the var. - async fn spawn_and_read_child_env( - vars: &[&str], - extra_env: &[(String, String)], - ) -> Vec { - let script = vars - .iter() - .map(|var| format!("printf '%s\\n' \"${{{var}:-}}\"")) - .collect::>() - .join("\n"); - let mut client = AcpClient::spawn("bash", &["-c".into(), script], extra_env, false) - .await - .expect("failed to spawn env probe script"); - let mut observed = Vec::with_capacity(vars.len()); - for var in vars { - observed.push( - client - .reader - .next() - .await - .unwrap_or_else(|| panic!("child produced no output for {var}")) - .expect("child stdout was not readable"), - ); - } - observed - } - - /// Every spawned agent must be told not to run the operator's cron - /// schedule, without the caller having to opt in. - #[tokio::test] - async fn spawn_injects_scheduler_disabled_env_by_default() { - let observed = spawn_and_read_child_env(&[GOOSE_SCHEDULER_DISABLED_ENV], &[]).await; - assert_eq!( - observed, - vec!["true"], - "{GOOSE_SCHEDULER_DISABLED_ENV} must be injected into every spawn" - ); - } - /// Spawn a probe script whose file name carries a runtime identity (e.g. /// `hermes-acp`) and return the value of `var` as the child observed it. /// `` means the child did not receive the var. @@ -2978,37 +2924,6 @@ mod tests { ); } - /// Persona config must not be able to re-enable the scheduler: this is a - /// correctness invariant, not an operator-tunable default, so the - /// injection is set after (and therefore wins over) the `extra_env` loop. - /// - /// The control var pins that `extra_env` really did reach the child, so a - /// pass here means the conflicting entry lost the fight rather than - /// `extra_env` being dropped wholesale. - #[tokio::test] - async fn spawn_scheduler_disabled_env_overrides_conflicting_extra_env() { - let extra_env = vec![ - ( - GOOSE_SCHEDULER_DISABLED_ENV.to_string(), - "false".to_string(), - ), - ( - "BUZZ_ENV_PROBE_CONTROL".to_string(), - "delivered".to_string(), - ), - ]; - let observed = spawn_and_read_child_env( - &[GOOSE_SCHEDULER_DISABLED_ENV, "BUZZ_ENV_PROBE_CONTROL"], - &extra_env, - ) - .await; - assert_eq!( - observed, - vec!["true", "delivered"], - "a persona extra_env entry must not override {GOOSE_SCHEDULER_DISABLED_ENV}" - ); - } - #[tokio::test] async fn idle_timeout_fires_on_silent_process() { let mut client = spawn_script("sleep 10").await; From 7e9b77f72d82e019a99f074f1c9829be30c57ae1 Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 11:36:12 -0600 Subject: [PATCH 14/32] Fix inline raster avatars in agent catalog (#3581) ## Summary - render existing shared personas whose catalog avatar is a bounded inline PNG, JPEG, GIF, or WebP data URL - keep rejecting arbitrary, malformed, unsupported, and oversized `data:` URLs - preserve hosted relay URLs as the forward format; catalog browsing remains read-only ## Root cause Paul's live shared kind:30175 head contains a 144,878-character `data:image/png;base64,...` avatar. The catalog projection accepted HTTP(S) URLs and bounded percent-encoded SVG emoji avatars only, so it projected Paul's avatar to `null` before `ProfileAvatar` rendered it. The owner still saw the local persona avatar, producing the reported owner/viewer mismatch. This patch accepts only four raster MIME types with strict base64 shape and a 256 KiB total URL cap at the existing catalog parsing boundary. It repairs already-signed heads such as Paul without viewer-side uploads or publication side effects. Hosted media remains the canonical forward path. #3578 uploads inline raster avatars during snapshot import, preventing the known source from creating future inline persona/profile values; existing signed catalog heads still need this compatibility path until their owners republish. ## Agent instruction finding The catalog publishes `AgentDefinition.system_prompt` verbatim as the user-authored **Agent instruction**, as documented by the sharing UI and NIP-AP. No Buzz base/core/runtime prompt is concatenated in the publish, catalog, or import path. This PR therefore does not remove authored instructions and accidentally strip copied agents of their behavior. ## Validation - targeted `personaCatalogRelay.test.mjs`: 24 passed - Desktop typecheck: passed - pre-push Desktop frontend suite: 3,771 passed - pre-push Desktop checks: passed - `git diff --check`: passed - independent review: no blockers, 9.4/10 Signed-off-by: Wes Co-authored-by: Carl --- .../agents/lib/personaCatalogRelay.test.mjs | 24 ++++++++++++++++++ .../agents/lib/personaCatalogRelay.ts | 25 ++++++++++++++++++- 2 files changed, 48 insertions(+), 1 deletion(-) diff --git a/desktop/src/features/agents/lib/personaCatalogRelay.test.mjs b/desktop/src/features/agents/lib/personaCatalogRelay.test.mjs index 24f6959b1c..fbaf1f5274 100644 --- a/desktop/src/features/agents/lib/personaCatalogRelay.test.mjs +++ b/desktop/src/features/agents/lib/personaCatalogRelay.test.mjs @@ -205,6 +205,30 @@ test("test_non_svg_data_avatar_is_rejected", () => { assert.equal(catalogAvatarUrl("data:image/png,%89PNG"), null); }); +test("test_legacy_inline_raster_avatar_survives_the_catalog", () => { + for (const mime of ["png", "jpeg", "gif", "webp"]) { + const avatar = `data:image/${mime};base64,iVBORw0KGgo=`; + assert.equal(catalogAvatarUrl(avatar), avatar); + } +}); + +test("test_inline_raster_avatar_rejects_unbounded_or_malformed_payloads", () => { + const prefix = "data:image/png;base64,"; + const payloadLength = 256 * 1_024 - prefix.length; + const validPayloadLength = payloadLength - (payloadLength % 4); + const withinCap = `${prefix}${"a".repeat(validPayloadLength - 2)}==`; + assert.ok(withinCap.length <= 256 * 1_024); + assert.equal(catalogAvatarUrl(withinCap), withinCap); + assert.equal( + catalogAvatarUrl( + `${withinCap}${"a".repeat(256 * 1_024 - withinCap.length + 1)}`, + ), + null, + ); + assert.equal(catalogAvatarUrl("data:image/png;base64,not base64"), null); + assert.equal(catalogAvatarUrl("data:image/bmp;base64,aA=="), null); +}); + test("test_oversized_inline_svg_avatar_is_rejected", () => { const withinCap = `data:image/svg+xml,${"a".repeat(8_192 - "data:image/svg+xml,".length)}`; assert.equal(withinCap.length, 8_192); diff --git a/desktop/src/features/agents/lib/personaCatalogRelay.ts b/desktop/src/features/agents/lib/personaCatalogRelay.ts index c85a976ba6..02c3f8e202 100644 --- a/desktop/src/features/agents/lib/personaCatalogRelay.ts +++ b/desktop/src/features/agents/lib/personaCatalogRelay.ts @@ -93,6 +93,16 @@ function isSafeHttpUrl(value: unknown): value is string { const INLINE_SVG_AVATAR_PREFIX = "data:image/svg+xml,"; const MAX_INLINE_SVG_AVATAR_LENGTH = 8_192; +/** + * Shared persona heads can carry an uploaded avatar as an inline raster. Keep + * those self-contained images renderable without accepting arbitrary `data:` + * URLs: only the raster MIME types browsers decode in ``, strict base64 + * shape, and a bound no larger than the relay's event-content ceiling. + */ +const MAX_INLINE_RASTER_AVATAR_LENGTH = 256 * 1_024; +const INLINE_RASTER_AVATAR_RE = + /^data:image\/(?:png|jpeg|gif|webp);base64,([A-Za-z0-9+/]+={0,2})$/u; + function isInlineSvgAvatar(value: unknown): value is string { return ( typeof value === "string" && @@ -101,6 +111,17 @@ function isInlineSvgAvatar(value: unknown): value is string { ); } +function isInlineRasterAvatar(value: unknown): value is string { + if ( + typeof value !== "string" || + value.length > MAX_INLINE_RASTER_AVATAR_LENGTH + ) { + return false; + } + const match = INLINE_RASTER_AVATAR_RE.exec(value); + return match !== null && (match[1]?.length ?? 0) % 4 === 0; +} + function optionalString(value: unknown): string | null { return typeof value === "string" && value.trim().length > 0 ? value : null; } @@ -121,7 +142,9 @@ function parsePersonaContent(event: RelayEvent): CatalogAgentProjection | null { } const avatarUrl = - isSafeHttpUrl(parsed.avatar_url) || isInlineSvgAvatar(parsed.avatar_url) + isSafeHttpUrl(parsed.avatar_url) || + isInlineSvgAvatar(parsed.avatar_url) || + isInlineRasterAvatar(parsed.avatar_url) ? parsed.avatar_url : null; const namePool = Array.isArray(parsed.name_pool) From 324bd6b464de5751e12abbd155376046ce3d2afc Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 12:03:20 -0600 Subject: [PATCH 15/32] Fix shared agent avatar import profiles (#3578) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit > Carl is updating this pull request on Wes's behalf. ## Summary - upload an embedded raster avatar through the existing authenticated media pipeline before minting or persisting an imported shared agent - store and publish only the resulting hosted URL so agent kind:0 profiles remain within content limits ## Root cause Snapshot import recovered raster avatar pixels as a large inline base64 data URL. That value was persisted and placed into the agent's kind:0 profile. The relay rejected the oversized profile, so other clients could not resolve the imported agent's avatar. ## Scope This is intentionally the forward fix only. It changes two Desktop files and does **not** add migration or reconciliation behavior for previously imported agents. Existing affected imports must be re-imported or fixed manually. ## Validation - successful pre-push Desktop suite: 1,863 passed, 14 ignored, 0 failed - all pre-push Rust/Desktop gates green, including all-target clippy - valid >256 KiB PNG import → production MIME detection/sanitization → bounded signed kind:0 containing only the hosted URL - upload failure, malformed data, and URL-only avatar cases covered - independent fresh review by Princess Donut: clean, no blocking findings Signed-off-by: Wes Co-authored-by: Carl --- desktop/src-tauri/src/commands/media.rs | 18 ++- .../src/commands/personas/snapshot/import.rs | 150 +++++++++++++++++- 2 files changed, 159 insertions(+), 9 deletions(-) diff --git a/desktop/src-tauri/src/commands/media.rs b/desktop/src-tauri/src/commands/media.rs index bf8692ff70..ed3b340238 100644 --- a/desktop/src-tauri/src/commands/media.rs +++ b/desktop/src-tauri/src/commands/media.rs @@ -411,7 +411,7 @@ fn should_retry_legacy_upload(status: reqwest::StatusCode) -> bool { } async fn send_upload_attempt( - state: &State<'_, AppState>, + state: &AppState, url: String, auth_header: &str, mime: &str, @@ -455,10 +455,22 @@ async fn send_upload_attempt( response.map_err(|error| classify_request_error(&error)) } +pub(crate) async fn upload_image_bytes( + body: Vec, + state: &AppState, +) -> Result { + let mime = detect_and_validate_mime(&body)?; + if !mime.starts_with("image/") { + return Err("profile avatar must be an image".to_string()); + } + let body = sanitize_image_for_upload(body, &mime)?; + do_upload(body, &mime, state, None).await +} + async fn do_upload( body: Vec, mime: &str, - state: &State<'_, AppState>, + state: &AppState, progress: Option<(tauri::AppHandle, String)>, ) -> Result { let sha256 = hex::encode(Sha256::digest(&body)); @@ -559,7 +571,7 @@ pub async fn upload_media( /// files from ever leaving the client on image-only surfaces. async fn process_picked_path( path: std::path::PathBuf, - state: &State<'_, AppState>, + state: &AppState, images_only: bool, ) -> Result { // Pin the inode by opening the fd BEFORE spawn_blocking. This prevents a diff --git a/desktop/src-tauri/src/commands/personas/snapshot/import.rs b/desktop/src-tauri/src/commands/personas/snapshot/import.rs index 7648941d23..d23efe7730 100644 --- a/desktop/src-tauri/src/commands/personas/snapshot/import.rs +++ b/desktop/src-tauri/src/commands/personas/snapshot/import.rs @@ -256,6 +256,24 @@ pub(crate) fn decode_snapshot_from_bytes( Ok(snapshot) } +async fn materialize_import_avatar( + avatar_data_url: Option<&str>, + avatar_url: Option<&str>, + upload: F, +) -> Result, String> +where + F: FnOnce(Vec) -> Fut, + Fut: std::future::Future>, +{ + let Some(avatar_data_url) = avatar_data_url else { + return Ok(avatar_url.map(str::to_string)); + }; + let avatar_bytes = + crate::managed_agents::agent_snapshot::decode_avatar_data_url(avatar_data_url) + .ok_or_else(|| "Snapshot avatar data is malformed.".to_string())?; + upload(avatar_bytes).await.map(Some) +} + // ── `preview_agent_snapshot_import` ────────────────────────────────────────── /// Decode and validate a snapshot file, returning a preview for the @@ -354,12 +372,21 @@ pub async fn confirm_agent_snapshot_import( )?; let minted_parallelism = minted.parallelism; - // Effective avatar: data URL wins; URL fallback when data URL is absent. - let effective_avatar: Option = snapshot - .profile - .avatar_data_url - .clone() - .or_else(|| snapshot.profile.avatar_url.clone()); + // Profile metadata must contain a hosted URL. Inline avatar data can be far + // larger than the relay's kind:0 content limit, so upload imported pixels + // before minting or persisting the new agent. Failing here keeps import + // atomic instead of creating an agent whose profile can never publish. + let effective_avatar = materialize_import_avatar( + snapshot.profile.avatar_data_url.as_deref(), + snapshot.profile.avatar_url.as_deref(), + |avatar_bytes| async { + crate::commands::media::upload_image_bytes(avatar_bytes, &state) + .await + .map(|descriptor| descriptor.url) + .map_err(|error| format!("Could not upload the imported avatar: {error}")) + }, + ) + .await?; // Wire-format string for the persona definition's respond_to field. // Omit when it is the default (owner-only) to keep definitions clean. @@ -711,3 +738,114 @@ async fn submit_engram_event( } Ok(()) } + +#[cfg(test)] +mod import_avatar_tests { + use super::materialize_import_avatar; + use std::cell::Cell; + + #[tokio::test] + async fn inline_avatar_is_uploaded_and_replaced_with_hosted_url() { + let uploaded = Cell::new(false); + let result = materialize_import_avatar( + Some("data:image/png;base64,iVBORw0KGgo="), + Some("https://sender.invalid/avatar.png"), + |bytes| { + uploaded.set(true); + async move { + assert_eq!(bytes, b"\x89PNG\r\n\x1a\n"); + Ok("https://relay.example/media/avatar.png".to_string()) + } + }, + ) + .await + .unwrap(); + + assert!(uploaded.get()); + assert_eq!( + result.as_deref(), + Some("https://relay.example/media/avatar.png") + ); + } + + #[tokio::test] + async fn hosted_avatar_skips_upload() { + let result = + materialize_import_avatar(None, Some("https://sender.example/avatar.png"), |_| async { + panic!("hosted avatars must not be uploaded") + }) + .await + .unwrap(); + + assert_eq!(result.as_deref(), Some("https://sender.example/avatar.png")); + } + + #[tokio::test] + async fn relay_sized_inline_avatar_becomes_bounded_signed_profile() { + use base64::{engine::general_purpose::STANDARD, Engine}; + use image::ImageEncoder; + use nostr::JsonUtil; + + let mut pixels = vec![0_u8; 512 * 512 * 4]; + let mut seed = 0x1234_5678_u32; + for byte in &mut pixels { + seed ^= seed << 13; + seed ^= seed >> 17; + seed ^= seed << 5; + *byte = seed as u8; + } + let mut source = Vec::new(); + image::codecs::png::PngEncoder::new(&mut source) + .write_image(&pixels, 512, 512, image::ExtendedColorType::Rgba8) + .unwrap(); + assert!(source.len() > 256 * 1024); + let data_url = format!("data:image/png;base64,{}", STANDARD.encode(&source)); + assert!(data_url.len() > 256 * 1024); + + let avatar = materialize_import_avatar(Some(&data_url), None, |bytes| async move { + let mime = crate::commands::media::detect_and_validate_mime(&bytes)?; + assert_eq!(mime, "image/png"); + let sanitized = crate::commands::media::sanitize_image_for_upload(bytes, &mime)?; + image::load_from_memory(&sanitized).map_err(|error| error.to_string())?; + Ok("https://relay.example/media/avatar.png".to_string()) + }) + .await + .unwrap() + .unwrap(); + + let event = + crate::events::build_profile(Some("Imported agent"), None, Some(&avatar), None, None) + .unwrap() + .sign_with_keys(&nostr::Keys::generate()) + .unwrap(); + assert!(event.content.len() < 64 * 1024); + assert!(!event.content.contains("data:image/")); + assert!(event + .content + .contains("https://relay.example/media/avatar.png")); + assert!(event.as_json().len() < 256 * 1024); + } + + #[tokio::test] + async fn upload_failure_aborts_avatar_materialization() { + let result = materialize_import_avatar( + Some("data:image/png;base64,iVBORw0KGgo="), + None, + |_| async { Err("relay upload failed".to_string()) }, + ) + .await; + + assert_eq!(result.unwrap_err(), "relay upload failed"); + } + + #[tokio::test] + async fn malformed_inline_avatar_fails_before_upload() { + let result = + materialize_import_avatar(Some("data:image/png;base64,not-base64!"), None, |_| async { + panic!("malformed avatars must not be uploaded") + }) + .await; + + assert_eq!(result.unwrap_err(), "Snapshot avatar data is malformed."); + } +} From 259de6afbe0cc0d106e57ebdb2323064990e4122 Mon Sep 17 00:00:00 2001 From: klopez4212 Date: Wed, 29 Jul 2026 19:49:57 +0100 Subject: [PATCH 16/32] Improve emoji autocomplete matching (#3571) ## Summary - Show all colon emoji autocomplete matches - Rank exact and prefix shortcodes before weaker matches - Add a regression test and screenshot ## Validation - `pnpm test` - `pnpm build` - `pnpm exec playwright test --project=smoke tests/e2e/custom-emoji.spec.ts --grep "exact standard shortcode"` - `just desktop-tauri-clippy` Native Tauri tests were attempted but could not link because the local disk filled during compilation. --------- Signed-off-by: kenny lopez --- .../messages/lib/useEmojiAutocomplete.ts | 38 +++++--- .../messages/ui/EmojiAutocomplete.tsx | 94 +++++++++++-------- desktop/src/shared/lib/emojiSearch.test.mjs | 30 ++++++ desktop/src/shared/lib/emojiSearch.ts | 27 ++++++ desktop/src/testing/e2eBridge.ts | 5 +- desktop/tests/e2e/custom-emoji.spec.ts | 52 ++++++++++ 6 files changed, 193 insertions(+), 53 deletions(-) diff --git a/desktop/src/features/messages/lib/useEmojiAutocomplete.ts b/desktop/src/features/messages/lib/useEmojiAutocomplete.ts index 53ef44b28d..57e458bc91 100644 --- a/desktop/src/features/messages/lib/useEmojiAutocomplete.ts +++ b/desktop/src/features/messages/lib/useEmojiAutocomplete.ts @@ -4,7 +4,11 @@ import { init, SearchIndex } from "emoji-mart"; import data from "@emoji-mart/data"; import type { CustomEmoji } from "@/shared/lib/remarkCustomEmoji"; -import { fuzzyStandardEmoji, rankByShortcode } from "@/shared/lib/emojiSearch"; +import { + fuzzyStandardEmoji, + rankByShortcode, + rankShortcodeMatchesFirst, +} from "@/shared/lib/emojiSearch"; import { rewriteRelayUrl } from "@/shared/lib/mediaUrl"; import type { AutocompleteEdit } from "./useRichTextEditor"; @@ -18,7 +22,7 @@ export type EmojiSuggestion = { const EMOJI_DEBOUNCE_MS = 120; const MIN_QUERY_LENGTH = 2; -const MAX_RESULTS = 8; +const UNLIMITED_RESULTS = Number.POSITIVE_INFINITY; init({ data }); @@ -81,7 +85,7 @@ export function useEmojiAutocomplete(customEmoji: CustomEmoji[] = []) { emojiQuery, customEmojiRef.current, (e) => e.shortcode, - MAX_RESULTS, + UNLIMITED_RESULTS, ).map((e) => ({ id: e.shortcode, name: e.shortcode, @@ -89,7 +93,10 @@ export function useEmojiAutocomplete(customEmoji: CustomEmoji[] = []) { url: rewriteRelayUrl(e.url), })); - SearchIndex.search(emojiQuery) + SearchIndex.search(emojiQuery, { + caller: "useEmojiAutocomplete", + maxResults: UNLIMITED_RESULTS, + }) .then( ( results: Array<{ @@ -106,23 +113,28 @@ export function useEmojiAutocomplete(customEmoji: CustomEmoji[] = []) { native: emoji.skins[0]?.native ?? "", })) .filter((e) => e.native !== ""); - // Top up remaining slots with fuzzy shortcode matches emoji-mart - // missed — its token-prefix search can't cross `_` (so `pointup` - // finds nothing). Skip ids already shown to avoid duplicates. + // Add fuzzy shortcode matches emoji-mart missed — its token-prefix + // search can't cross `_` (so `pointup` finds nothing). Skip ids + // already shown to avoid duplicates. const shown = new Set( [...customMatches, ...standard].map((e) => e.id), ); const fuzzy: EmojiSuggestion[] = fuzzyStandardEmoji( emojiQuery, - MAX_RESULTS - customMatches.length - standard.length, + UNLIMITED_RESULTS, shown, ).map((e) => ({ id: e.id, name: e.name, native: e.native })); - // Custom emoji first (community-specific), then standard, then fuzzy. - const merged = [...customMatches, ...standard, ...fuzzy].slice( - 0, - MAX_RESULTS, + // Rank exact/prefix shortcode matches across custom and standard emoji + // before semantic and weaker matches (for example, `joy` before + // `bufo_joy`). Keep emoji-mart's name/keyword results ahead of loose + // substring and subsequence matches. + setSuggestions( + rankShortcodeMatchesFirst( + emojiQuery, + [...standard, ...customMatches, ...fuzzy], + (emoji) => emoji.id, + ), ); - setSuggestions(merged); setEmojiSelectedIndex(0); }, ) diff --git a/desktop/src/features/messages/ui/EmojiAutocomplete.tsx b/desktop/src/features/messages/ui/EmojiAutocomplete.tsx index 3b6fa3a2d2..d44933aef1 100644 --- a/desktop/src/features/messages/ui/EmojiAutocomplete.tsx +++ b/desktop/src/features/messages/ui/EmojiAutocomplete.tsx @@ -2,6 +2,10 @@ import * as React from "react"; import type { EmojiSuggestion } from "@/features/messages/lib/useEmojiAutocomplete"; import { cn } from "@/shared/lib/cn"; +import { + type ListVirtualizer, + VirtualizedList, +} from "@/shared/ui/VirtualizedList"; import { POPOVER_CUSTOM_ENTER_MOTION_CLASS, POPOVER_SHADOW_STYLE, @@ -21,15 +25,21 @@ export const EmojiAutocomplete = React.memo(function EmojiAutocomplete({ onSelect, position = "above", }: EmojiAutocompleteProps) { - const listRef = React.useRef(null); + const listVirtualizerRef = React.useRef(null); React.useEffect(() => { - const activeItem = listRef.current?.children[selectedIndex] as - | HTMLElement - | undefined; - activeItem?.scrollIntoView({ block: "nearest" }); + listVirtualizerRef.current?.scrollToIndex(selectedIndex, { + align: "auto", + }); }, [selectedIndex]); + const handleVirtualizer = React.useCallback( + (virtualizer: ListVirtualizer) => { + listVirtualizerRef.current = virtualizer; + }, + [], + ); + if (suggestions.length === 0) { return null; } @@ -43,47 +53,55 @@ export const EmojiAutocomplete = React.memo(function EmojiAutocomplete({ >
- {suggestions.map((suggestion, index) => ( - - ))} + suggestion.id} + items={suggestions} + onVirtualizer={handleVirtualizer} + renderItem={(suggestion, index) => ( + + )} + />
); diff --git a/desktop/src/shared/lib/emojiSearch.test.mjs b/desktop/src/shared/lib/emojiSearch.test.mjs index 93cb1e286a..01a52293f5 100644 --- a/desktop/src/shared/lib/emojiSearch.test.mjs +++ b/desktop/src/shared/lib/emojiSearch.test.mjs @@ -5,6 +5,7 @@ import { fuzzyStandardEmoji, normalizeShortcode, rankByShortcode, + rankShortcodeMatchesFirst, scoreShortcodeMatch, } from "./emojiSearch.ts"; @@ -77,6 +78,35 @@ test("rankByShortcode respects the limit", () => { assert.equal(ranked.length, 2); }); +test("exact shortcode matches rank ahead of weaker custom shortcode matches", () => { + const items = [ + { code: "bufo_joy", source: "custom" }, + { code: "joy", source: "standard" }, + { code: "joy_cat", source: "custom" }, + { code: "face_with_tears_of_joy", source: "standard" }, + ]; + const ranked = rankShortcodeMatchesFirst("joy", items, (item) => item.code); + + assert.deepEqual( + ranked.map((item) => item.code), + ["joy", "joy_cat", "bufo_joy", "face_with_tears_of_joy"], + ); +}); + +test("semantic results stay ahead of loose shortcode matches", () => { + const items = [ + { code: "frowning_face", source: "semantic" }, + { code: "sandwich", source: "custom" }, + { code: "sad", source: "standard" }, + ]; + const ranked = rankShortcodeMatchesFirst("sad", items, (item) => item.code); + + assert.deepEqual( + ranked.map((item) => item.code), + ["sad", "frowning_face", "sandwich"], + ); +}); + test("fuzzyStandardEmoji surfaces point_up for `pointup`", () => { const hits = fuzzyStandardEmoji("pointup", 8, new Set()); const ids = hits.map((e) => e.id); diff --git a/desktop/src/shared/lib/emojiSearch.ts b/desktop/src/shared/lib/emojiSearch.ts index 2a0d74b9bb..ec4ea529b5 100644 --- a/desktop/src/shared/lib/emojiSearch.ts +++ b/desktop/src/shared/lib/emojiSearch.ts @@ -114,6 +114,33 @@ export function rankByShortcode( return scored.slice(0, limit).map((s) => s.item); } +/** + * Place exact and prefix shortcode matches ahead of items that only matched an + * emoji name or keyword. This lets an exact standard emoji like `joy` beat a + * weaker custom shortcode match such as `bufo_joy`, while retaining emoji-mart's + * order for name- and keyword-only results ahead of loose shortcode matches. + */ +export function rankShortcodeMatchesFirst( + query: string, + items: readonly T[], + shortcodeOf: (item: T) => string, +): T[] { + const strongShortcodeMatches = rankByShortcode( + query, + items, + shortcodeOf, + Number.POSITIVE_INFINITY, + ).filter((item) => { + const match = scoreShortcodeMatch(query, shortcodeOf(item)); + return match !== null && match.tier <= TIER_PREFIX; + }); + const matchedItems = new Set(strongShortcodeMatches); + return [ + ...strongShortcodeMatches, + ...items.filter((item) => !matchedItems.has(item)), + ]; +} + export interface StandardEmoji { id: string; name: string; diff --git a/desktop/src/testing/e2eBridge.ts b/desktop/src/testing/e2eBridge.ts index 7b13273c60..4cfc553df1 100644 --- a/desktop/src/testing/e2eBridge.ts +++ b/desktop/src/testing/e2eBridge.ts @@ -913,8 +913,8 @@ function createMockRelayMembershipEvent(): RelayEvent { * sets from distinct pubkeys so the e2e exercises the union/collapse path, not * a single relay-owned set. `:buzz:` is the stable shortcode exercised by * custom-emoji.spec.ts (claimed by BOTH members with different URLs, so the - * palette must collapse it to one deterministic winner); `:narf:` proves a - * second member's distinct emoji unions in. + * palette must collapse it to one deterministic winner); `:narf:` and + * `:bufo_joy:` prove a second member's distinct emoji unions in. */ function createMockCustomEmojiSetEvents(): RelayEvent[] { return [ @@ -941,6 +941,7 @@ function createMockCustomEmojiSetEvents(): RelayEvent[] { // member B claims :buzz: with a DIFFERENT url — unionCustomEmoji must // collapse it to one deterministic winner, never expose two URLs. ["emoji", "buzz", "https://example.com/e2e/buzz-b.png"], + ["emoji", "bufo_joy", "https://example.com/e2e/bufo-joy.png"], ], "b".repeat(64), ), diff --git a/desktop/tests/e2e/custom-emoji.spec.ts b/desktop/tests/e2e/custom-emoji.spec.ts index aa345570f9..ae20aaef25 100644 --- a/desktop/tests/e2e/custom-emoji.spec.ts +++ b/desktop/tests/e2e/custom-emoji.spec.ts @@ -1,6 +1,9 @@ import { expect, test } from "@playwright/test"; +import * as fs from "node:fs"; +import * as path from "node:path"; import { installMockBridge } from "../helpers/bridge"; +import { waitForAnimations } from "../helpers/animations"; // Custom-emoji end-to-end guard. // @@ -77,6 +80,55 @@ test("typing a known :shortcode: renders an inline emoji node in the composer", await expect(input).not.toContainText(`:${SHORTCODE}:`); }); +test("emoji autocomplete ranks an exact standard shortcode before a custom substring", async ({ + page, +}, testInfo) => { + await openGeneral(page); + + const input = page.getByTestId("message-input"); + await input.click(); + await input.pressSequentially(":joy"); + + const autocomplete = page.getByTestId("emoji-autocomplete"); + await expect(autocomplete).toBeVisible(); + const labels = await autocomplete.locator("button").allTextContents(); + expect(labels[0]).toContain(":joy:"); + expect( + labels.findIndex((label) => label.includes(":bufo_joy:")), + ).toBeGreaterThan(0); + + const screenshotDir = path.resolve( + "test-results/emoji-autocomplete-screenshots", + ); + fs.mkdirSync(screenshotDir, { recursive: true }); + const screenshotPath = path.join( + screenshotDir, + "joy-exact-before-custom-substring.png", + ); + await waitForAnimations(page); + await autocomplete.screenshot({ path: screenshotPath }); + await testInfo.attach("joy-exact-before-custom-substring", { + path: screenshotPath, + contentType: "image/png", + }); +}); + +test("emoji autocomplete keeps semantic matches ahead of loose shortcode fallbacks", async ({ + page, +}) => { + await openGeneral(page); + + const input = page.getByTestId("message-input"); + await input.click(); + await input.pressSequentially(":sad"); + + const autocomplete = page.getByTestId("emoji-autocomplete"); + await expect(autocomplete).toBeVisible(); + await expect(autocomplete.locator("button").first()).not.toContainText( + ":sandwich:", + ); +}); + test("custom emoji deletes as a single unit (like a built-in emoji)", async ({ page, }) => { From 047533c56c2a2d03f23ef3edb990e58405767aac Mon Sep 17 00:00:00 2001 From: Krishna C Date: Wed, 29 Jul 2026 15:02:42 -0400 Subject: [PATCH 17/32] fix(mobile): keep TLS on relays joined by invite (#3139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Communities joined via an invite link never connect: the app dials `ws://` on port 80 instead of `wss://` on 443 and sits on "Reconnecting…" indefinitely. `RelayConfig.baseUrl` is documented as an HTTP origin, but the two onboarding flows disagree on what they persist: - **Device pairing** validates and stores `https://` — `pairing_provider.dart:657` throws on anything else. - **Invite join** stores the relay URL straight off the invite link, and `deep_link.dart:165` always emits `ws://` or `wss://`. `wsUrl` only special-cased `https://`, so a `wss://` base fell through to the plaintext branch: ```dart final scheme = uri.scheme == 'https' ? 'wss' : 'ws'; // 'wss' is not 'https' ``` The claim request itself succeeds, because `_claimUrlFromRelay` (`invite_join_provider.dart:242`) maps `wss → https` explicitly. Only the socket path is missing that conversion — which is why the community appears, correctly named, and then never loads. The same `baseUrl` also feeds `/query` (`relay_session.dart:136`), media upload (`media_upload.dart:765`), Blossom auth (`media_auth.dart:128`) and `relayClientProvider` (`relay_provider.dart:113`), so those requests were malformed too. Where port 80 *does* answer, it is additionally a silent TLS downgrade after `validateInviteRelayUri` insisted on `wss://`. This folds the websocket schemes back to their HTTP equivalents in `baseUrl` itself, so every consumer is correct by construction rather than needing a second getter remembered at each call site, and communities **already persisted** with `wss://` are repaired on read without a migration. `community_icon_provider.dart:46` already performs this same conversion locally. One subtlety worth flagging for review: the normalization is derived in the getter rather than applied in the constructor, so the constructor stays `const`. The compile-time fallback at `relay_provider.dart:77` relies on const canonicalization for a stable identity across rebuilds, and Riverpod's `defaultUpdateShouldNotify` is `previous != next` (`element.dart:361`), which falls back to identity for this class. A `factory` constructor here yields a fresh instance per rebuild, which tears down and resubscribes every listener — `channels_provider_test.dart` catches it as an unexpected unsubscribe during reconnect. ### Related issue Fixes #2662. ### Testing `flutter test` — **705 passed, 1 skipped, 0 failed** `flutter analyze` — No issues found `dart format --set-exit-if-changed .` — 249 files, 0 changed Run against the Hermit-pinned SDK (Flutter 3.41.7 / Dart 3.11.5), matching CI. 10 new unit tests in `mobile/test/shared/relay/relay_config_test.dart` covering both onboarding schemes, `http`/`https` passthrough, non-default ports, and agreement between the invite and pairing paths for the same relay. Verified end-to-end against a self-hosted relay behind `tailscale serve`, which terminates TLS on 443 and leaves port 80 closed. Relay logs show the invite claim succeeding over HTTPS at the moment of joining, while no WebSocket connection ever arrives — no `WebSocket connection established`, no NIP-42 auth, no `kind:0` profile, no push registration — across the relay's entire history, even though the member row is present and correct. Port-80 refusals are not logged by `tailscaled`'s netstack, which is why the retries leave no trace server-side. Reproduced on both iOS and Android. --------- Signed-off-by: Krishna C --- .../activity/compose_drafts_provider.dart | 8 +- .../search/recent_searches_provider.dart | 10 +- .../shared/relay/identity_scoped_prefs.dart | 50 ++++++++++ mobile/lib/shared/relay/relay.dart | 1 + mobile/lib/shared/relay/relay_provider.dart | 38 +++++++- .../compose_drafts_provider_test.dart | 29 ++++++ .../search/recent_searches_provider_test.dart | 21 +++++ .../relay/identity_scoped_prefs_test.dart | 92 +++++++++++++++++++ .../test/shared/relay/relay_config_test.dart | 67 ++++++++++++++ 9 files changed, 312 insertions(+), 4 deletions(-) create mode 100644 mobile/lib/shared/relay/identity_scoped_prefs.dart create mode 100644 mobile/test/shared/relay/identity_scoped_prefs_test.dart create mode 100644 mobile/test/shared/relay/relay_config_test.dart diff --git a/mobile/lib/features/activity/compose_drafts_provider.dart b/mobile/lib/features/activity/compose_drafts_provider.dart index b9a079e9a8..b5755f19c8 100644 --- a/mobile/lib/features/activity/compose_drafts_provider.dart +++ b/mobile/lib/features/activity/compose_drafts_provider.dart @@ -79,7 +79,13 @@ class ComposeDraftsNotifier extends Notifier> { _prefsKey = '$_draftsPrefsKey:${config.baseUrl}:$pubkey'; final prefs = ref.read(savedPrefsProvider); - final raw = prefs.getString(_prefsKey); + final raw = readMigratedPref( + prefs, + canonicalKey: _prefsKey, + legacyKey: '$_draftsPrefsKey:${config.storedOrigin}:$pubkey', + read: prefs.getString, + write: prefs.setString, + ); if (raw == null) return const []; try { final decoded = jsonDecode(raw); diff --git a/mobile/lib/features/search/recent_searches_provider.dart b/mobile/lib/features/search/recent_searches_provider.dart index 813672a9cc..2fad17832e 100644 --- a/mobile/lib/features/search/recent_searches_provider.dart +++ b/mobile/lib/features/search/recent_searches_provider.dart @@ -19,8 +19,16 @@ class RecentSearchesNotifier extends Notifier> { final pubkey = ref.watch(myPubkeyProvider) ?? 'anon'; _prefsKey = '$_recentSearchesPrefsKey:${config.baseUrl}:$pubkey'; + final prefs = ref.read(savedPrefsProvider); final stored = - ref.read(savedPrefsProvider).getStringList(_prefsKey) ?? const []; + readMigratedPref>( + prefs, + canonicalKey: _prefsKey, + legacyKey: '$_recentSearchesPrefsKey:${config.storedOrigin}:$pubkey', + read: prefs.getStringList, + write: prefs.setStringList, + ) ?? + const []; return List.unmodifiable( stored .map((query) => query.trim()) diff --git a/mobile/lib/shared/relay/identity_scoped_prefs.dart b/mobile/lib/shared/relay/identity_scoped_prefs.dart new file mode 100644 index 0000000000..e080b1ca8c --- /dev/null +++ b/mobile/lib/shared/relay/identity_scoped_prefs.dart @@ -0,0 +1,50 @@ +import 'dart:async'; + +import 'package:shared_preferences/shared_preferences.dart'; + +/// Reads an identity-scoped preference, migrating values left under a +/// pre-canonicalization relay origin. +/// +/// Identity-scoped keys embed the relay origin, and that origin used to be +/// whatever scheme the onboarding flow happened to persist — `wss://` for an +/// invite join, `https://` for device pairing. Now that [RelayConfig.baseUrl] +/// canonicalizes the scheme, an install that joined by invite would compute a +/// different key than the one its data was written under, leaving that data on +/// disk but unreachable. +/// +/// The value under [canonicalKey] wins. Otherwise a value under [legacyKey] is +/// returned straight away and promoted onto the canonical key in the +/// background. The legacy entry is removed only once the copy reports success, +/// so a migration interrupted mid-flight leaves the original readable and the +/// next read simply retries. +/// +/// Pass matching accessors for the stored type — `getString`/`setString`, or +/// `getStringList`/`setStringList`. +T? readMigratedPref( + SharedPreferences prefs, { + required String canonicalKey, + required String legacyKey, + required T? Function(String key) read, + required Future Function(String key, T value) write, +}) { + final canonical = read(canonicalKey); + if (canonical != null) return canonical; + + // Pairing-created communities already store an HTTP origin, so the two keys + // coincide and there is nothing to migrate. + if (canonicalKey == legacyKey) return null; + + final legacy = read(legacyKey); + if (legacy == null) return null; + + unawaited(_promote(prefs, legacyKey, write(canonicalKey, legacy))); + return legacy; +} + +Future _promote( + SharedPreferences prefs, + String legacyKey, + Future copy, +) async { + if (await copy) await prefs.remove(legacyKey); +} diff --git a/mobile/lib/shared/relay/relay.dart b/mobile/lib/shared/relay/relay.dart index d168b9a097..bc11325414 100644 --- a/mobile/lib/shared/relay/relay.dart +++ b/mobile/lib/shared/relay/relay.dart @@ -1,4 +1,5 @@ export 'app_lifecycle_provider.dart'; +export 'identity_scoped_prefs.dart'; export 'media_auth.dart'; export 'media_image.dart'; export 'media_upload.dart'; diff --git a/mobile/lib/shared/relay/relay_provider.dart b/mobile/lib/shared/relay/relay_provider.dart index 97b88dd3df..061dd6cb38 100644 --- a/mobile/lib/shared/relay/relay_provider.dart +++ b/mobile/lib/shared/relay/relay_provider.dart @@ -10,12 +10,46 @@ import 'relay_client.dart'; /// - `baseUrl` — where the relay lives (used for WS + media upload) /// - `nsec` — the user's signing key (drives NIP-42 AUTH and event sigs) class RelayConfig { - final String baseUrl; + const RelayConfig({required String baseUrl, this.nsec}) : _baseUrl = baseUrl; + + /// Relay origin exactly as the active community stored it. + final String _baseUrl; /// Nostr secret key (bech32 nsec) for signing events and NIP-42 AUTH. final String? nsec; - const RelayConfig({required this.baseUrl, this.nsec}); + /// The origin as persisted, before scheme canonicalization. + /// + /// Exists solely so identity-scoped storage keys written before [baseUrl] + /// was canonicalized stay reachable — see [readMigratedPref]. Never use it + /// for network I/O; [baseUrl] and [wsUrl] are the addresses to connect to. + String get storedOrigin => _baseUrl; + + /// Relay origin as an HTTP(S) URL. + /// + /// Communities are persisted with whichever scheme their onboarding flow + /// used: device pairing stores `https://` (it rejects anything else), while + /// an invite join stores the `wss://` relay URL carried by the invite link. + /// Every consumer treats this as an HTTP origin — [wsUrl], the `/query` + /// endpoint, media upload and Blossom auth — so a `wss://` base silently + /// degrades all of them. Folding the websocket schemes back here keeps both + /// onboarding paths equivalent, including for already-persisted communities. + /// + /// Derived rather than normalized in the constructor so that the constructor + /// stays `const`: the compile-time fallback below relies on canonicalization + /// to keep its identity stable across rebuilds, and Riverpod's default + /// `updateShouldNotify` is `previous != next`, which falls back to identity + /// here. A fresh instance per rebuild would resubscribe every listener. + String get baseUrl { + final uri = Uri.tryParse(_baseUrl); + if (uri == null) return _baseUrl; + final scheme = switch (uri.scheme) { + 'wss' => 'https', + 'ws' => 'http', + _ => null, + }; + return scheme == null ? _baseUrl : uri.replace(scheme: scheme).toString(); + } /// Derive the websocket URL from the HTTP base URL. String get wsUrl { diff --git a/mobile/test/features/activity/compose_drafts_provider_test.dart b/mobile/test/features/activity/compose_drafts_provider_test.dart index 1f93dcb674..15899e6597 100644 --- a/mobile/test/features/activity/compose_drafts_provider_test.dart +++ b/mobile/test/features/activity/compose_drafts_provider_test.dart @@ -34,6 +34,35 @@ void main() { return container; } + test( + 'an invite-joined community keeps its drafts after origin canonicalization', + () async { + // Written by a build that stored the invite link's wss:// origin + // verbatim; RelayConfig now canonicalizes that to https://, so the key + // the app computes no longer matches the key on disk. + const legacyKey = 'compose_drafts_v1:wss://relay-a.example:pk_a'; + const canonicalKey = 'compose_drafts_v1:https://relay-a.example:pk_a'; + SharedPreferences.setMockInitialValues({ + legacyKey: + '[{"key":"ch1","channel_id":"ch1","text":"unsent work",' + '"updated_at":1700000000}]', + }); + + final container = await containerWithPrefs( + relayUrl: 'wss://relay-a.example', + ); + + final drafts = container.read(composeDraftsProvider); + expect(drafts, hasLength(1), reason: 'draft survives the upgrade'); + expect(drafts.single.text, 'unsent work'); + + await Future.delayed(Duration.zero); + final prefs = await SharedPreferences.getInstance(); + expect(prefs.getString(canonicalKey), isNotNull); + expect(prefs.getString(legacyKey), isNull); + }, + ); + test('composeDraftKey separates channel and thread composers', () { expect(composeDraftKey('ch1'), 'ch1'); expect(composeDraftKey('ch1', threadHeadId: 't1'), 'ch1:t1'); diff --git a/mobile/test/features/search/recent_searches_provider_test.dart b/mobile/test/features/search/recent_searches_provider_test.dart index 30f13c329c..df0a08a8ea 100644 --- a/mobile/test/features/search/recent_searches_provider_test.dart +++ b/mobile/test/features/search/recent_searches_provider_test.dart @@ -35,6 +35,27 @@ void main() { return container; } + test('an invite-joined community keeps its recent searches after ' + 'origin canonicalization', () async { + const legacyKey = 'recent_searches_v1:wss://relay-a.example:pk-a'; + const canonicalKey = 'recent_searches_v1:https://relay-a.example:pk-a'; + SharedPreferences.setMockInitialValues({ + legacyKey: ['nostr', 'relays'], + }); + + final container = await containerWithPrefs( + relayUrl: 'wss://relay-a.example', + pubkey: 'pk-a', + ); + + expect(container.read(recentSearchesProvider), ['nostr', 'relays']); + + await Future.delayed(Duration.zero); + final prefs = await SharedPreferences.getInstance(); + expect(prefs.getStringList(canonicalKey), ['nostr', 'relays']); + expect(prefs.getStringList(legacyKey), isNull); + }); + test( 'normalizes, deduplicates, caps, and persists submitted queries', () async { diff --git a/mobile/test/shared/relay/identity_scoped_prefs_test.dart b/mobile/test/shared/relay/identity_scoped_prefs_test.dart new file mode 100644 index 0000000000..4fce3f254b --- /dev/null +++ b/mobile/test/shared/relay/identity_scoped_prefs_test.dart @@ -0,0 +1,92 @@ +import 'package:buzz/shared/relay/identity_scoped_prefs.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:shared_preferences/shared_preferences.dart'; + +void main() { + TestWidgetsFlutterBinding.ensureInitialized(); + + const canonical = 'k_v1:https://relay.example:pk'; + const legacy = 'k_v1:wss://relay.example:pk'; + + Future prefsWith(Map values) async { + SharedPreferences.setMockInitialValues(values); + return SharedPreferences.getInstance(); + } + + String? readString(SharedPreferences prefs) => readMigratedPref( + prefs, + canonicalKey: canonical, + legacyKey: legacy, + read: prefs.getString, + write: prefs.setString, + ); + + test('returns the canonical value when present', () async { + final prefs = await prefsWith({canonical: 'new', legacy: 'old'}); + expect(readString(prefs), 'new'); + }); + + test('falls back to the legacy value and returns it immediately', () async { + final prefs = await prefsWith({legacy: 'carried over'}); + expect(readString(prefs), 'carried over'); + }); + + test('promotes the legacy value onto the canonical key', () async { + final prefs = await prefsWith({legacy: 'carried over'}); + readString(prefs); + await Future.delayed(Duration.zero); + + expect(prefs.getString(canonical), 'carried over'); + expect(prefs.getString(legacy), isNull, reason: 'legacy entry is cleared'); + }); + + test('is idempotent across repeated reads', () async { + final prefs = await prefsWith({legacy: 'carried over'}); + readString(prefs); + await Future.delayed(Duration.zero); + expect(readString(prefs), 'carried over'); + await Future.delayed(Duration.zero); + expect(prefs.getString(canonical), 'carried over'); + }); + + test('returns null when neither key holds a value', () async { + final prefs = await prefsWith({}); + expect(readString(prefs), isNull); + }); + + test('does not touch storage when the keys coincide', () async { + // A pairing-created community already stores an HTTP origin, so canonical + // and legacy are the same string and there is nothing to migrate. + SharedPreferences.setMockInitialValues({canonical: 'only'}); + final prefs = await SharedPreferences.getInstance(); + final value = readMigratedPref( + prefs, + canonicalKey: canonical, + legacyKey: canonical, + read: prefs.getString, + write: prefs.setString, + ); + await Future.delayed(Duration.zero); + + expect(value, 'only'); + expect(prefs.getString(canonical), 'only'); + }); + + test('migrates string lists as well as strings', () async { + final prefs = await prefsWith({ + legacy: ['alpha', 'beta'], + }); + final value = readMigratedPref>( + prefs, + canonicalKey: canonical, + legacyKey: legacy, + read: prefs.getStringList, + write: prefs.setStringList, + ); + await Future.delayed(Duration.zero); + + expect(value, ['alpha', 'beta']); + expect(prefs.getStringList(canonical), ['alpha', 'beta']); + expect(prefs.getStringList(legacy), isNull); + }); +} diff --git a/mobile/test/shared/relay/relay_config_test.dart b/mobile/test/shared/relay/relay_config_test.dart new file mode 100644 index 0000000000..d0decb3685 --- /dev/null +++ b/mobile/test/shared/relay/relay_config_test.dart @@ -0,0 +1,67 @@ +import 'package:flutter_test/flutter_test.dart'; + +import 'package:buzz/shared/relay/relay_provider.dart'; + +void main() { + group('RelayConfig.baseUrl normalization', () { + test('folds a wss:// community URL to https://', () { + // Invite joins persist the relay URL straight off the invite link, which + // deep_link.dart always emits as ws:// or wss://. + final config = RelayConfig(baseUrl: 'wss://relay.example.com'); + expect(config.baseUrl, 'https://relay.example.com'); + }); + + test('folds a ws:// community URL to http://', () { + final config = RelayConfig(baseUrl: 'ws://relay.example.com:3000'); + expect(config.baseUrl, 'http://relay.example.com:3000'); + }); + + test('leaves an https:// community URL untouched', () { + // Device pairing rejects anything but https://, so these already conform. + final config = RelayConfig(baseUrl: 'https://relay.example.com'); + expect(config.baseUrl, 'https://relay.example.com'); + }); + + test('leaves an http:// community URL untouched', () { + final config = RelayConfig(baseUrl: 'http://localhost:3000'); + expect(config.baseUrl, 'http://localhost:3000'); + }); + + test('preserves a non-default port', () { + final config = RelayConfig(baseUrl: 'wss://relay.example.com:8443'); + expect(config.baseUrl, 'https://relay.example.com:8443'); + }); + }); + + group('RelayConfig.wsUrl', () { + test('keeps TLS for a relay joined by invite', () { + // Regression: a wss:// base used to fall through to the non-https branch + // and downgrade to ws://, dialing port 80 — which never connects on a + // relay that only serves 443, and drops TLS everywhere else. + final config = RelayConfig(baseUrl: 'wss://relay.example.com'); + expect(config.wsUrl, 'wss://relay.example.com'); + }); + + test('keeps TLS for a relay added by pairing', () { + final config = RelayConfig(baseUrl: 'https://relay.example.com'); + expect(config.wsUrl, 'wss://relay.example.com'); + }); + + test('both onboarding paths agree on the same relay', () { + final invited = RelayConfig(baseUrl: 'wss://relay.example.com'); + final paired = RelayConfig(baseUrl: 'https://relay.example.com'); + expect(invited.wsUrl, paired.wsUrl); + expect(invited.baseUrl, paired.baseUrl); + }); + + test('stays plaintext for local development', () { + final config = RelayConfig(baseUrl: 'http://localhost:3000'); + expect(config.wsUrl, 'ws://localhost:3000'); + }); + + test('preserves a non-default port', () { + final config = RelayConfig(baseUrl: 'wss://relay.example.com:8443'); + expect(config.wsUrl, 'wss://relay.example.com:8443'); + }); + }); +} From b42a8d447e3a2b85b2313dc4fdd123731fd8bba3 Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 13:46:47 -0600 Subject: [PATCH 18/32] fix(desktop): reconcile thread arrivals at bottom (#3585) ## Summary - reconcile stale native-scroll anchors when a reply arrives at the physical floor - clear the thread new-message affordance instead of incrementing it from stale cached state - preserve the existing mid-history path and add direct lifecycle regression coverage ## Why PR #3411 fixed geometry-driven reconciliation, but the reply-arrival branch still trusted a cached `message` anchor without checking the rendered position. Native anchoring could return a short thread to the floor without another scroll/resize callback, then the next reply incremented the pill anyway. ## Verification - Desktop checks passed - Desktop typecheck passed - focused lifecycle test passed (6/6) - push hook full Desktop unit suite passed (3,770/3,770) - `git diff --check` passed Signed-off-by: Wes Co-authored-by: Carl --- .../ui/useAnchoredScroll.lifecycle.test.mjs | 93 ++++++++++++++++++- .../features/messages/ui/useAnchoredScroll.ts | 16 ++++ 2 files changed, 108 insertions(+), 1 deletion(-) diff --git a/desktop/src/features/messages/ui/useAnchoredScroll.lifecycle.test.mjs b/desktop/src/features/messages/ui/useAnchoredScroll.lifecycle.test.mjs index 1fdfb05857..ee3fec1a98 100644 --- a/desktop/src/features/messages/ui/useAnchoredScroll.lifecycle.test.mjs +++ b/desktop/src/features/messages/ui/useAnchoredScroll.lifecycle.test.mjs @@ -247,13 +247,20 @@ function Harness({ channelId, onTargetSettled, refs }) { return null; } -function BottomStateHarness({ messages, onState, refs }) { +function BottomStateHarness({ + messages, + onState, + refs, + targetMessageId = null, +}) { const anchored = useAnchoredScroll({ channelId: "conversation", contentRef: refs.content, isLoading: false, messages, + pinTargetCentered: targetMessageId !== null, scrollContainerRef: refs.container, + targetMessageId, }); onState(anchored); return null; @@ -329,6 +336,90 @@ test("channel change attaches pinned-center observers after refs mount", async ( }); }); +test("arrival at the physical floor does not preserve a stale unread state", async () => { + const refs = { + container: { current: null }, + content: { current: null }, + }; + const root = createRoot(document.createElement("div")); + const nodes = makePinnedCenterNodes(); + refs.container.current = nodes.container; + refs.content.current = nodes.content; + let state = null; + const render = (messages) => + root.render( + React.createElement(BottomStateHarness, { + messages, + onState: (nextState) => { + state = nextState; + }, + refs, + }), + ); + + await act(async () => render([{ id: "first" }])); + await act(async () => new Promise((resolve) => setTimeout(resolve, 0))); + nodes.container.scrollTop = 100; + await act(async () => state.onScroll()); + nodes.container.scrollTop = 100; + await act(async () => state.onScroll()); + assert.equal(state.isAtBottom, false); + + // Native anchoring can return the viewport to the floor without a scroll or + // resize callback, leaving only the hook's cached message anchor stale. + nodes.container.scrollTop = + nodes.container.scrollHeight - nodes.container.clientHeight; + await act(async () => render([{ id: "first" }, { id: "second" }])); + + assert.equal(state.isAtBottom, true); + assert.equal(state.newMessageCount, 0); + await act(async () => root.unmount()); +}); + +test("arrival does not steal an active layout target during floor-like reflow", async () => { + const refs = { + container: { current: null }, + content: { current: null }, + }; + const root = createRoot(document.createElement("div")); + const nodes = makePinnedCenterNodes(); + refs.container.current = nodes.container; + refs.content.current = nodes.content; + let state = null; + const render = (messages, targetMessageId = null) => + root.render( + React.createElement(BottomStateHarness, { + messages, + onState: (nextState) => { + state = nextState; + }, + refs, + targetMessageId, + }), + ); + + await act(async () => render([{ id: "selected" }])); + await act(async () => new Promise((resolve) => setTimeout(resolve, 0))); + nodes.container.scrollTop = 100; + await act(async () => state.onScroll()); + nodes.container.scrollTop = 100; + await act(async () => state.onScroll()); + assert.equal(state.isAtBottom, false); + + // A focus/split presentation switch can commit fresh replies while the old + // container geometry momentarily reads as the physical floor. The explicit + // layout target must win so the reading row is restored after reflow. + nodes.container.scrollTop = + nodes.container.scrollHeight - nodes.container.clientHeight; + await act(async () => + render([{ id: "selected" }, { id: "second" }], "selected"), + ); + + assert.equal(state.isAtBottom, false); + assert.equal(state.newMessageCount, 1); + await act(async () => root.unmount()); +}); + test("container resize clears a stale new-message state at the physical floor", async () => { const refs = { container: { current: null }, diff --git a/desktop/src/features/messages/ui/useAnchoredScroll.ts b/desktop/src/features/messages/ui/useAnchoredScroll.ts index add9439599..0bfcb3b3e2 100644 --- a/desktop/src/features/messages/ui/useAnchoredScroll.ts +++ b/desktop/src/features/messages/ui/useAnchoredScroll.ts @@ -714,6 +714,22 @@ export function useAnchoredScroll({ container.scrollTo({ top: container.scrollHeight, behavior: "auto" }); } if (newLatestArrived) setNewMessageCount(0); + } else if ( + messagesArrived > 0 && + !targetMessageId && + !virtualizerOwnsPrependAnchoring && + isAtBottomNow(container) + ) { + // A native scroll/layout callback may not have reconciled a stale + // message anchor before this append commits. If the rendered result is + // still physically at the floor (common in short threads), do not turn + // that stale anchor into a visible unread affordance. Active navigation + // targets own the viewport and must be preserved across presentation + // reflow even when the old geometry momentarily reads as the floor. + anchorRef.current = { kind: "at-bottom" }; + container.scrollTo({ top: container.scrollHeight, behavior: "auto" }); + setIsAtBottom(true); + setNewMessageCount(0); } else if (messagesArrived > 0 && !virtualizerOwnsPrependAnchoring) { // Anchored mid-history. An older-history prepend grows the content above // the reading row; the browser's native scroll anchoring does NOT correct From 66e7054928cc29395f828467c3e8c81b7408dd29 Mon Sep 17 00:00:00 2001 From: Logan Johnson Date: Wed, 29 Jul 2026 15:48:06 -0400 Subject: [PATCH 19/32] fix(desktop): deduplicate relay outage notification (#3579) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🤖 ## Summary - Keep the relay reconnect notification dismissed across repeated connection retries during one continuous outage. - Re-arm the notification after recovery or relay lifecycle replacement, including switches between communities that use the same relay URL. - Preserve the existing dedicated path for authentication and other application-level errors. ### How an outage is tracked The AppShell-owned relay-card hook treats an outage as one contiguous runtime episode rather than assigning it a persisted ID. A hook-local `outageActiveRef` is armed by the first qualifying unreachable/degraded observation. While it is armed, intermediate retry states (`connecting`, `reconnecting`, `stalled`, and `disconnected`) belong to that same episode, so retry churn cannot clear dismissal or emit another notification. The hook receives the same lifecycle identity used by community initialization: community ID plus `reinitKey`. This distinguishes multiple communities even when they share a relay URL, and it also changes when the active community is explicitly reinitialized. The latch and dismissal are reset when that identity changes or when the relay singleton reports its authoritative `idle` teardown state. A successful `connected` state also closes the episode and re-arms the next outage. These boundaries deliberately bias toward re-notifying rather than suppressing a later outage: recovery, community switch/reinit, or relay teardown cannot leave the hook stuck believing an old outage is still active. No outage state is persisted beyond the mounted hook lifecycle. ### Related issue None found. ### Testing - `pnpm --dir desktop typecheck` - `pnpm --dir desktop test` — 3,769 passed - `pnpm --dir desktop check` - `pnpm --dir desktop build:e2e` - `pnpm --dir desktop exec playwright test tests/e2e/sidebar-relay-card.spec.ts --project=integration` — 11 passed --------- Signed-off-by: Larry <8cf5a83f590ec0955b11647d1c88f796a98e088c30a492c58e0e46c3026ae7a4@buzz.block.builderlab.xyz> Co-authored-by: npub13n66s06epmqf2kc3v373ez8hj65cuzyvxzjf93vwpervxqn2u7jq2qd9je <8cf5a83f590ec0955b11647d1c88f796a98e088c30a492c58e0e46c3026ae7a4@buzz.block.builderlab.xyz> --- desktop/src/app/AppShell.tsx | 1 + .../ui/useSidebarRelayConnectionCard.ts | 35 +++++++--- desktop/tests/e2e/sidebar-relay-card.spec.ts | 68 +++++++++++++++++++ 3 files changed, 93 insertions(+), 11 deletions(-) diff --git a/desktop/src/app/AppShell.tsx b/desktop/src/app/AppShell.tsx index 75f57257cc..abbbf29610 100644 --- a/desktop/src/app/AppShell.tsx +++ b/desktop/src/app/AppShell.tsx @@ -231,6 +231,7 @@ export function AppShell() { const relayConnectionCard = useSidebarRelayConnectionCard( channelsErrorMessage, communitiesHook.activeCommunity?.relayUrl, + `${communitiesHook.activeCommunity?.id ?? "none"}-${communitiesHook.reinitKey}`, ); const memberChannels = React.useMemo( () => channels.filter((channel) => channel.isMember), diff --git a/desktop/src/features/sidebar/ui/useSidebarRelayConnectionCard.ts b/desktop/src/features/sidebar/ui/useSidebarRelayConnectionCard.ts index f92e1a02e8..11f2bd1064 100644 --- a/desktop/src/features/sidebar/ui/useSidebarRelayConnectionCard.ts +++ b/desktop/src/features/sidebar/ui/useSidebarRelayConnectionCard.ts @@ -63,6 +63,7 @@ function isDocumentVisible() { export function useSidebarRelayConnectionCard( errorMessage?: string, relayUrl?: string | null, + relayLifecycleKey = relaySuccessKey(relayUrl), ) { const relayConnectionState = useRelayConnection(); const hasRelayUnreachableError = errorMessage @@ -79,7 +80,6 @@ export function useSidebarRelayConnectionCard( relayConnectionState === "stalled" || (relayConnectionState === "disconnected" && !hasNonUnreachableError); const isRelayConnectionConnected = relayConnectionState === "connected"; - const isRelayConnectionDisconnected = relayConnectionState === "disconnected"; const [isDismissed, setIsDismissed] = React.useState(false); const hasSuccess = React.useSyncExternalStore( subscribeRelayConnectivitySuccess, @@ -95,6 +95,8 @@ export function useSidebarRelayConnectionCard( const isRelayConnectionSuccess = hasSuccess && isRelayConnectionConnected; const canShow = isRelayConnectionActuallyDegraded || isRelayConnectionSuccess; const show = canShow && !isDismissed; + const outageActiveRef = React.useRef(false); + const outageRelayLifecycleKeyRef = React.useRef(relayLifecycleKey); const wasProblemCardVisibleRef = React.useRef(false); const { isPending: isReconnectPending, @@ -111,30 +113,41 @@ export function useSidebarRelayConnectionCard( isReconnectPending || connectivityAction === "relay-connection"; React.useEffect(() => { - if (!isRelayConnectionActuallyDegraded && !isRelayConnectionSuccess) { + if (outageRelayLifecycleKeyRef.current !== relayLifecycleKey) { + outageRelayLifecycleKeyRef.current = relayLifecycleKey; + outageActiveRef.current = false; + wasProblemCardVisibleRef.current = false; setIsDismissed(false); } - }, [isRelayConnectionSuccess, isRelayConnectionActuallyDegraded]); - React.useEffect(() => { - if (isRelayConnectionStateDegraded || isRelayConnectionDisconnected) { - setRelayConnectivitySuccess(relayUrl, false); + if (relayConnectionState === "idle") { + outageActiveRef.current = false; + wasProblemCardVisibleRef.current = false; setIsDismissed(false); + return; } - }, [isRelayConnectionDisconnected, isRelayConnectionStateDegraded, relayUrl]); - React.useEffect(() => { if (isRelayConnectionActuallyDegraded) { + if (!outageActiveRef.current) { + outageActiveRef.current = true; + setRelayConnectivitySuccess(relayUrl, false); + setIsDismissed(false); + } wasProblemCardVisibleRef.current = show && !isRelayConnectionSuccess; return; } - if (wasProblemCardVisibleRef.current && isRelayConnectionConnected) { - wasProblemCardVisibleRef.current = false; - setRelayConnectivitySuccess(relayUrl, true); + if (outageActiveRef.current && isRelayConnectionConnected) { + outageActiveRef.current = false; + if (wasProblemCardVisibleRef.current) { + wasProblemCardVisibleRef.current = false; + setRelayConnectivitySuccess(relayUrl, true); + } } }, [ isRelayConnectionSuccess, + relayLifecycleKey, + relayConnectionState, relayUrl, show, isRelayConnectionActuallyDegraded, diff --git a/desktop/tests/e2e/sidebar-relay-card.spec.ts b/desktop/tests/e2e/sidebar-relay-card.spec.ts index 9505c0f10f..b9903614d2 100644 --- a/desktop/tests/e2e/sidebar-relay-card.spec.ts +++ b/desktop/tests/e2e/sidebar-relay-card.spec.ts @@ -84,6 +84,25 @@ async function setRelayConnectionState( }, state); } +async function emitRelayConnectionState( + page: Page, + state: RelayConnectionState, +) { + await page.evaluate((nextState) => { + const setConnectionState = ( + window as Window & { + __BUZZ_E2E_SET_RELAY_CONNECTION_STATE__?: ( + state: RelayConnectionState, + ) => void; + } + ).__BUZZ_E2E_SET_RELAY_CONNECTION_STATE__; + if (!setConnectionState) { + throw new Error("Mock relay connection state helper is not installed."); + } + setConnectionState(nextState); + }, state); +} + async function expectGenericReconnectCard(page: Page) { const card = page.getByTestId("sidebar-relay-unreachable"); await expect(card).toBeVisible(); @@ -109,6 +128,55 @@ test("sidebar generic relay failures use the reconnect card", async ({ await expectGenericReconnectCard(page); }); +test("relay outage notification stays dismissed through retries and re-arms after recovery", async ({ + page, +}) => { + await installMockBridge(page, { channelsReadError: CONNECT_ERROR }); + await page.goto("/"); + await setRelayConnectionState(page, "disconnected"); + + const card = await expectGenericReconnectCard(page); + await card + .getByRole("button", { name: "Dismiss relay notification" }) + .click({ force: true }); + await expect(card).toBeHidden(); + + // Retry churn is still the same outage: no successful connection occurred. + await emitRelayConnectionState(page, "connecting"); + await emitRelayConnectionState(page, "disconnected"); + await emitRelayConnectionState(page, "reconnecting"); + await page.waitForTimeout(2_100); + await expect(card).toBeHidden(); + + // A successful connection ends the episode and re-arms the next outage. + await setChannelsReadError(page, null); + await emitRelayConnectionState(page, "connected"); + await setChannelsReadError(page, CONNECT_ERROR); + await emitRelayConnectionState(page, "disconnected"); + await expectGenericReconnectCard(page); +}); + +test("relay outage notification re-arms after same-URL lifecycle teardown", async ({ + page, +}) => { + await installMockBridge(page, { channelsReadError: CONNECT_ERROR }); + await page.goto("/"); + await setRelayConnectionState(page, "disconnected"); + + const card = await expectGenericReconnectCard(page); + await card + .getByRole("button", { name: "Dismiss relay notification" }) + .click({ force: true }); + await expect(card).toBeHidden(); + + // Community switches and reconnectCommunity() tear down the singleton to + // idle before applying the next lifecycle. The next lifecycle may reuse the + // same relay URL, so URL identity alone must not preserve the old dismissal. + await emitRelayConnectionState(page, "idle"); + await emitRelayConnectionState(page, "disconnected"); + await expectGenericReconnectCard(page); +}); + test("sidebar proxy sign-in failures use the reconnect card", async ({ page, }) => { From 7adc46268d5e93f0b1d4dc8e700af22815dcac1b Mon Sep 17 00:00:00 2001 From: Logan Johnson Date: Wed, 29 Jul 2026 16:37:53 -0400 Subject: [PATCH 20/32] feat(cli): mirror Desktop mention delivery (#3330) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🤖 ## Summary Agent-authored mentions currently depend on matching visible `@Name` text to channel profiles. That makes notification delivery ambiguous when names collide or profiles change, and it encourages an extra post-send lookup just to confirm that the intended `p` tags were emitted. This change makes `buzz messages send` mirror Desktop's existing model: the message keeps a readable name in its content while the recipient pubkey is supplied separately. ```bash buzz messages send \ --channel \ --content '@Alice could you review this?' \ --mention ``` `--mention` is repeatable. The CLI normalizes and deduplicates explicit pubkeys, merges them with any names it can resolve from the channel, and gives explicit identities priority under the existing 50-mention limit. Before uploading attachments, signing, or publishing, the command checks every resulting pubkey against the channel's current membership: - Members are mentioned normally. - Non-members stop the send and produce an actionable error. - `--allow-non-member-mentions` deliberately sends notifying `p` tags without adding anyone to the channel. Sending a message never changes membership. On success, `mention_pubkeys` is read from the exact signed event and returned with the relay response, so callers can verify the emitted recipients without another query. Managed-agent guidance teaches this single-command mention flow. Desktop mention behavior and the Nostr event schema are unchanged. Forum guidance is intentionally handled separately in #3596. ### Related issue None found. This replaces the earlier guidance-only approach in this PR with the underlying CLI behavior it required. ### Testing - `cargo test -p buzz-sdk` - `cargo test -p buzz-cli` - `cargo test -p buzz-acp` - `cargo test --manifest-path desktop/src-tauri/Cargo.toml` --------- Signed-off-by: npub1fdupjvyregj3z2tx7gx5x6py04zw89jm5usef9lyea4f3vcgh8qq9zgkdz <4b78193083ca25112966f20d4368247d44e3965ba7219497e4cf6a98b308b9c0@buzz.block.builderlab.xyz> Signed-off-by: npub13n66s06epmqf2kc3v373ez8hj65cuzyvxzjf93vwpervxqn2u7jq2qd9je <8cf5a83f590ec0955b11647d1c88f796a98e088c30a492c58e0e46c3026ae7a4@buzz.block.builderlab.xyz> Co-authored-by: npub1fdupjvyregj3z2tx7gx5x6py04zw89jm5usef9lyea4f3vcgh8qq9zgkdz <4b78193083ca25112966f20d4368247d44e3965ba7219497e4cf6a98b308b9c0@buzz.block.builderlab.xyz> Co-authored-by: npub13n66s06epmqf2kc3v373ez8hj65cuzyvxzjf93vwpervxqn2u7jq2qd9je <8cf5a83f590ec0955b11647d1c88f796a98e088c30a492c58e0e46c3026ae7a4@buzz.block.builderlab.xyz> --- crates/buzz-acp/src/base_prompt.md | 2 + crates/buzz-acp/src/lib.rs | 16 + crates/buzz-cli/src/commands/messages.rs | 294 +++++++++++++++--- crates/buzz-cli/src/lib.rs | 3 + desktop/src-tauri/src/managed_agents/nest.rs | 2 +- .../src/managed_agents/nest/tests.rs | 12 + .../src/managed_agents/nest_skill.md | 9 +- 7 files changed, 288 insertions(+), 50 deletions(-) diff --git a/crates/buzz-acp/src/base_prompt.md b/crates/buzz-acp/src/base_prompt.md index c42e65cb83..e360d24982 100644 --- a/crates/buzz-acp/src/base_prompt.md +++ b/crates/buzz-acp/src/base_prompt.md @@ -40,6 +40,8 @@ For explicit changes to an existing personal agent, use `buzz agents draft-updat - Use the person's **exact full display name** after `@` (e.g., `@Will Pfleger`, not `@Will`). Partial names fail silently. - Do NOT format mentions with bold, italic, or backticks — it breaks notification delivery. +- When you know intended recipient pubkeys, send readable `@Name` text and pass the identities separately in the same command: `buzz messages send ... --content "@Name ..." --mention `. Repeat `--mention` for multiple recipients. Any explicit identity (`--mention` or `nostr:npub...`) permits unresolved or ambiguous `@Name` text as presentation-only; uniquely resolved member names still add their own recipients. Include a pubkey for every presentation-only name that should notify. The success JSON's `mention_pubkeys` comes from the signed event and is the delivery evidence; no follow-up verification command is needed. +- Without `--mention`, the CLI resolves `@Name` against current channel members. It stops before sending on an unresolved/ambiguous name or a mentioned pubkey that is not a member. For a non-member, add them explicitly with `buzz channels add-member` only when authorized, then retry. Sending never changes membership automatically. - Only `@mention` when you need their attention. Don't mention in narrative (e.g., "coordinating with Duncan" — no `@`). Naming someone while talking *about* them is narrative — "waiting on @morgan", "until @morgan brings work", "I'll loop in @morgan later". Drop the `@`. Every mention sends a notification; a mention nobody needs to act on is a false alarm. ### Callback Mentions diff --git a/crates/buzz-acp/src/lib.rs b/crates/buzz-acp/src/lib.rs index d63f720c65..403512a322 100644 --- a/crates/buzz-acp/src/lib.rs +++ b/crates/buzz-acp/src/lib.rs @@ -3625,6 +3625,22 @@ mod agent_draft_prompt_tests { assert!(prompt.contains("single-quoted shell strings preserve `\\n` literally")); assert!(prompt.contains("buzz messages send ... --content -")); } + + #[test] + fn shared_base_prompt_teaches_single_command_mentions_and_preflight() { + let prompt = include_str!("base_prompt.md"); + assert!(prompt.contains("--mention ")); + assert!(prompt.contains("every presentation-only name that should notify")); + assert!( + prompt.contains("permits unresolved or ambiguous `@Name` text as presentation-only") + ); + assert!(prompt.contains("success JSON's `mention_pubkeys`")); + assert!(prompt.contains("no follow-up verification command is needed")); + assert!(prompt.contains("stops before sending")); + assert!(prompt + .contains("add them explicitly with `buzz channels add-member` only when authorized")); + assert!(prompt.contains("never changes membership automatically")); + } } fn default_heartbeat_prompt() -> String { diff --git a/crates/buzz-cli/src/commands/messages.rs b/crates/buzz-cli/src/commands/messages.rs index 290cc59fa8..40a9ae80b5 100644 --- a/crates/buzz-cli/src/commands/messages.rs +++ b/crates/buzz-cli/src/commands/messages.rs @@ -9,8 +9,7 @@ use crate::validate::{ validate_content_size, validate_hex64, validate_uuid, MAX_DIFF_BYTES, }; use buzz_sdk::mentions::{ - extract_at_mentions_with_known, extract_nostr_uris, merge_mentions, strip_code_regions, - MENTION_CAP, + extract_at_mentions_with_known, extract_nostr_uris, strip_code_regions, MENTION_CAP, }; /// Extract the thread root event ID from a Nostr tag array. @@ -119,47 +118,82 @@ async fn resolve_channel_id(client: &BuzzClient, event_id: &str) -> Result>, + has_explicit_mentions: bool, +) -> Result, CliError> { + let mut resolved = Vec::new(); + for name in names { + match name_to_pubkeys + .get(name) + .map(Vec::as_slice) + .unwrap_or_default() + { + [pubkey] => resolved.push(pubkey.clone()), + [] if has_explicit_mentions => {} + [] => { + return Err(CliError::Usage(format!( + "mention '@{name}' does not match a current channel member; retry with --mention " + ))) + } + _ if has_explicit_mentions => {} + candidates => { + return Err(CliError::Usage(format!( + "mention '@{name}' is ambiguous; candidates: {}. Retry with --mention ", + candidates.join(", ") + ))) + } + } + } + Ok(resolved) +} + +/// Resolve mention text against the channel membership snapshot. /// -/// Queries kind 39002 (channel members) then kind 0 (profiles), parses -/// display names once, and feeds them to [`extract_at_mentions_with_known`] -/// for multi-word matching. On any I/O or parse failure, returns an empty -/// vec — auto-tagging is best-effort and must never block a send. +/// Returns both the current member set and uniquely name-resolved pubkeys. +/// Lookup failures are fatal when mention processing is requested: publishing +/// visible mention text without its intended `p` tag is worse than not sending. async fn resolve_content_mentions( client: &BuzzClient, channel_id: &str, content: &str, -) -> Vec { - if !content.contains('@') { - return vec![]; + has_explicit_mentions: bool, +) -> Result<(Vec, Vec), CliError> { + let stripped = strip_code_regions(content); + if !stripped.contains('@') && !has_explicit_mentions { + return Ok((vec![], vec![])); } - // 1. Membership list (kind 39002 is parameterized-replaceable, addressed by `d` tag). let members_filter = serde_json::json!({ "kinds": [39002], "#d": [channel_id], "limit": 1, }); - let member_pubkeys = match fetch_member_pubkeys(client, &members_filter).await { - Some(pks) if !pks.is_empty() => pks, - _ => return vec![], - }; + let member_pubkeys = fetch_member_pubkeys(client, &members_filter) + .await + .ok_or_else(|| { + CliError::Other("could not load channel membership for mention preflight".into()) + })?; + + if !stripped.contains('@') { + return Ok((member_pubkeys, vec![])); + } - // 2. Profiles for those members (kind 0). let profiles_filter = serde_json::json!({ "kinds": [0], "authors": member_pubkeys, "limit": member_pubkeys.len(), }); - let profile_events = match fetch_events(client, &profiles_filter).await { - Some(v) => v, - None => return vec![], - }; + let profile_events = fetch_events(client, &profiles_filter) + .await + .ok_or_else(|| { + CliError::Other("could not load member profiles for mention resolution".into()) + })?; - // 3. Single parse: extract (pubkey, display_name) pairs from profile JSON. let mut name_to_pubkeys: std::collections::HashMap> = std::collections::HashMap::new(); - let mut display_names: Vec = Vec::new(); + let mut display_names = Vec::new(); for e in &profile_events { let Some(pubkey) = e.get("pubkey").and_then(|v| v.as_str()) else { continue; @@ -178,26 +212,82 @@ async fn resolve_content_mentions( else { continue; }; - let lower = name.to_ascii_lowercase(); name_to_pubkeys - .entry(lower) + .entry(name.to_ascii_lowercase()) .or_default() .push(pubkey.to_string()); display_names.push(name.to_string()); } - // 4. Two-pass extraction: known multi-word names first, single-word fallback. - let known_refs: Vec<&str> = display_names.iter().map(|s| s.as_str()).collect(); - let names = extract_at_mentions_with_known(content, &known_refs); + let known_refs: Vec<&str> = display_names.iter().map(String::as_str).collect(); + let names = extract_at_mentions_with_known(&stripped, &known_refs); + let resolved = resolve_names_to_pubkeys(&names, &name_to_pubkeys, has_explicit_mentions)?; + Ok((member_pubkeys, resolved)) +} + +fn normalize_explicit_mentions(values: &[String]) -> Result, CliError> { + let mut normalized = Vec::new(); + for value in values { + let pubkey = PublicKey::parse(value.trim()) + .map_err(|_| CliError::Usage(format!("invalid --mention pubkey: {value}")))?; + let hex = pubkey.to_hex(); + if !normalized.contains(&hex) { + normalized.push(hex); + } + } + if normalized.len() > MENTION_CAP { + return Err(CliError::Usage(format!( + "too many --mention values (max {MENTION_CAP})" + ))); + } + Ok(normalized) +} + +fn merge_message_mentions( + explicit: &[String], + uri_pubkeys: &[String], + auto_resolved: &[String], +) -> Result, CliError> { + let mut mentions = Vec::new(); + for pubkey in explicit + .iter() + .chain(uri_pubkeys.iter()) + .chain(auto_resolved.iter()) + { + if !mentions.contains(pubkey) { + mentions.push(pubkey.clone()); + } + } + if mentions.len() > MENTION_CAP { + return Err(CliError::Usage(format!( + "too many unique message mentions (max {MENTION_CAP})" + ))); + } + Ok(mentions) +} - // 5. Look up matched names → pubkeys via the map we already built. - names +fn missing_members(mentions: &[String], members: &[String]) -> Vec { + let members: std::collections::HashSet<&str> = members.iter().map(String::as_str).collect(); + mentions .iter() - .flat_map(|n| name_to_pubkeys.get(n).into_iter().flatten()) + .filter(|pk| !members.contains(pk.as_str())) .cloned() .collect() } +fn event_mention_pubkeys(event: &nostr::Event) -> Vec { + event + .tags + .iter() + .filter_map(|tag| { + let parts = tag.as_slice(); + (parts.first().map(String::as_str) == Some("p")) + .then(|| parts.get(1).cloned()) + .flatten() + }) + .collect() +} + /// Fetch raw events for `filter` via the relay's `/query` endpoint. /// Returns `None` on any I/O or parse failure. async fn fetch_events( @@ -478,6 +568,7 @@ pub struct SendMessageParams { pub reply_to: Option, pub broadcast: bool, pub files: Vec, + pub mentions: Vec, } pub async fn cmd_send_message( @@ -495,6 +586,30 @@ pub async fn cmd_send_message( } let channel_uuid = parse_uuid(&p.channel_id)?; + let explicit_mentions = normalize_explicit_mentions(&p.mentions)?; + let stripped = strip_code_regions(&p.content); + let uri_pubkeys = extract_nostr_uris(&stripped); + // Supplying any identity explicitly authorizes unresolved or ambiguous @Name text + // as presentation-only, matching Desktop's separate visible-label and p-tag model. + // Uniquely resolvable member names still add their own p-tags; callers must supply + // every intended identity whose visible label cannot be resolved uniquely. + let has_explicit_mentions = !explicit_mentions.is_empty() || !uri_pubkeys.is_empty(); + let (member_pubkeys, auto_resolved) = + resolve_content_mentions(client, &p.channel_id, &p.content, has_explicit_mentions).await?; + let mention_pubkeys = merge_message_mentions(&explicit_mentions, &uri_pubkeys, &auto_resolved)?; + + let missing = missing_members(&mention_pubkeys, &member_pubkeys); + if !missing.is_empty() { + return Err(CliError::Usage( + serde_json::json!({ + "message": "mentioned pubkeys are not channel members; add them explicitly before retrying", + "missing_member_pubkeys": missing, + "add_member_command": format!("buzz channels add-member --channel {} --pubkey --role ", p.channel_id), + }) + .to_string(), + )); + } + // Upload files and build imeta tags let mut media_tags: Vec> = Vec::new(); let mut media_content = String::new(); @@ -526,16 +641,7 @@ pub async fn cmd_send_message( None }; - // Resolve @name mentions in the author-written body only — not the media markdown we - // append above, which is derived from upload metadata and can't carry `@names`. - let mut auto_resolved = resolve_content_mentions(client, &p.channel_id, &p.content).await; - - // NIP-27: also extract nostr:npub1… inline references (skipping code regions) - let stripped = strip_code_regions(&p.content); - let uri_pubkeys = extract_nostr_uris(&stripped); - merge_mentions(&mut auto_resolved, &uri_pubkeys, MENTION_CAP); - - let mention_refs: Vec<&str> = auto_resolved.iter().map(|s| s.as_str()).collect(); + let mention_refs: Vec<&str> = mention_pubkeys.iter().map(String::as_str).collect(); let builder = match p.kind { Some(45001) => { @@ -572,9 +678,17 @@ pub async fn cmd_send_message( }; let event = client.sign_event(builder)?; - + let emitted_mentions = event_mention_pubkeys(&event); let resp = client.submit_event(event).await?; - println!("{}", normalize_write_response(&resp)); + let mut output: serde_json::Value = serde_json::from_str(&normalize_write_response(&resp)) + .unwrap_or_else(|_| serde_json::json!({ "response": resp })); + if let Some(object) = output.as_object_mut() { + object.insert( + "mention_pubkeys".into(), + serde_json::json!(emitted_mentions), + ); + } + println!("{output}"); Ok(()) } @@ -765,6 +879,7 @@ pub async fn dispatch( reply_to, broadcast, files, + mentions, } => { cmd_send_message( client, @@ -775,6 +890,7 @@ pub async fn dispatch( reply_to, broadcast, files, + mentions, }, ) .await @@ -876,7 +992,11 @@ pub async fn dispatch( #[cfg(test)] mod tests { - use super::{find_root_from_tags, match_profiles_by_name, parse_member_pubkeys}; + use super::{ + event_mention_pubkeys, find_root_from_tags, match_profiles_by_name, merge_message_mentions, + missing_members, normalize_explicit_mentions, parse_member_pubkeys, + resolve_names_to_pubkeys, + }; use buzz_sdk::mentions::{ extract_at_mentions_with_known, extract_at_names, match_names_to_profiles, MentionProfile, }; @@ -1103,6 +1223,94 @@ mod tests { assert_eq!(parse_member_pubkeys(&event), vec![PK_VALID_A, PK_VALID_A]); } + #[test] + fn explicit_mentions_accept_hex_and_npub_and_deduplicate() { + use nostr::ToBech32; + let npub = nostr::PublicKey::from_hex(PK_VALID_A) + .unwrap() + .to_bech32() + .unwrap(); + assert_eq!( + normalize_explicit_mentions(&[PK_VALID_A.into(), npub]).unwrap(), + vec![PK_VALID_A] + ); + assert!(normalize_explicit_mentions(&["not-a-key".into()]).is_err()); + } + + #[test] + fn explicit_mentions_authorize_presentation_text_without_name_resolution() { + let names = vec!["renamed user".into()]; + let profiles = std::collections::HashMap::new(); + assert_eq!( + resolve_names_to_pubkeys(&names, &profiles, true).unwrap(), + Vec::::new() + ); + assert!(resolve_names_to_pubkeys(&names, &profiles, false).is_err()); + } + + #[test] + fn explicit_mentions_authorize_ambiguous_presentation_text() { + let names = vec!["alice".into()]; + let profiles = std::collections::HashMap::from([( + "alice".into(), + vec![PK_VALID_A.into(), PK_VALID_B.into()], + )]); + assert_eq!( + resolve_names_to_pubkeys(&names, &profiles, true).unwrap(), + Vec::::new() + ); + let error = resolve_names_to_pubkeys(&names, &profiles, false).unwrap_err(); + assert!(error.to_string().contains(PK_VALID_A)); + assert!(error.to_string().contains(PK_VALID_B)); + } + + #[test] + fn explicit_mentions_make_all_at_names_presentation_only() { + let names = vec!["alice".into(), "bob".into()]; + let profiles = std::collections::HashMap::from([("alice".into(), vec![PK_VALID_A.into()])]); + assert_eq!( + resolve_names_to_pubkeys(&names, &profiles, true).unwrap(), + vec![PK_VALID_A] + ); + assert!(resolve_names_to_pubkeys(&names, &profiles, false).is_err()); + } + + #[test] + fn combined_mention_union_errors_instead_of_truncating() { + let explicit: Vec = (0..50).map(|i| format!("explicit-{i}")).collect(); + assert!(merge_message_mentions(&explicit, &[], &["resolved-bob".into()]).is_err()); + + let mut with_duplicate = explicit.clone(); + with_duplicate.push(explicit[0].clone()); + assert_eq!( + merge_message_mentions(&with_duplicate, &[explicit[1].clone()], &[]) + .unwrap() + .len(), + 50 + ); + } + + #[test] + fn membership_preflight_lists_only_missing_mentions() { + assert_eq!( + missing_members( + &[PK_VALID_A.into(), PK_VALID_B.into()], + &[PK_VALID_A.into()] + ), + vec![PK_VALID_B] + ); + } + + #[test] + fn mention_evidence_comes_from_signed_event_tags() { + use nostr::{EventBuilder, Keys, Tag}; + let event = EventBuilder::text_note("hello") + .tags(vec![Tag::parse(["p", PK_VALID_A]).unwrap()]) + .sign_with_keys(&Keys::generate()) + .unwrap(); + assert_eq!(event_mention_pubkeys(&event), vec![PK_VALID_A]); + } + // ---- match_profiles_by_name (author resolution for `messages search --author`) ---- fn profile_event( diff --git a/crates/buzz-cli/src/lib.rs b/crates/buzz-cli/src/lib.rs index 7465625804..df02c65be9 100644 --- a/crates/buzz-cli/src/lib.rs +++ b/crates/buzz-cli/src/lib.rs @@ -369,6 +369,9 @@ pub enum MessagesCmd { /// Attach file(s) — uploads and includes as imeta tags #[arg(long = "file")] files: Vec, + /// Pubkey to mention (hex or npub; repeatable). Supplying any explicit identity permits unresolved or ambiguous @Name text as presentation-only; uniquely resolved member names still notify. + #[arg(long = "mention")] + mentions: Vec, }, /// Send a code diff / patch to a channel SendDiff { diff --git a/desktop/src-tauri/src/managed_agents/nest.rs b/desktop/src-tauri/src/managed_agents/nest.rs index e13ab2baad..c8f008836d 100644 --- a/desktop/src-tauri/src/managed_agents/nest.rs +++ b/desktop/src-tauri/src/managed_agents/nest.rs @@ -50,7 +50,7 @@ const NEST_AGENTS_VERSION: u32 = 4; /// Template content version for SKILL.md. /// Bump this when changing `nest_skill.md` to trigger refresh on existing installs. -const NEST_SKILL_VERSION: u32 = 4; +const NEST_SKILL_VERSION: u32 = 5; const BEGIN_MARKER: &str = ""; diff --git a/desktop/src-tauri/src/managed_agents/nest/tests.rs b/desktop/src-tauri/src/managed_agents/nest/tests.rs index d2c415e725..031b049a49 100644 --- a/desktop/src-tauri/src/managed_agents/nest/tests.rs +++ b/desktop/src-tauri/src/managed_agents/nest/tests.rs @@ -29,6 +29,18 @@ fn init_nest_dir_prod_sets_buzz() { } } +#[test] +fn nest_skill_contains_safe_mention_workflow() { + assert!(BUZZ_CLI_SKILL_MD.contains("--mention ")); + assert!(BUZZ_CLI_SKILL_MD.contains("every presentation-only name that should notify")); + assert!(BUZZ_CLI_SKILL_MD + .contains("permits unresolved or ambiguous `@Name` text as presentation-only")); + assert!(BUZZ_CLI_SKILL_MD.contains("signed event's `mention_pubkeys`")); + assert!(BUZZ_CLI_SKILL_MD.contains("no follow-up verification command is needed")); + assert!(BUZZ_CLI_SKILL_MD.contains("Add membership separately only when authorized")); + assert!(BUZZ_CLI_SKILL_MD.contains("never changes membership automatically")); +} + #[test] fn ensure_nest_creates_all_dirs_and_agents_md() { let tmp = tempfile::tempdir().unwrap(); diff --git a/desktop/src-tauri/src/managed_agents/nest_skill.md b/desktop/src-tauri/src/managed_agents/nest_skill.md index fefdfa77fa..79a5ea301d 100644 --- a/desktop/src-tauri/src/managed_agents/nest_skill.md +++ b/desktop/src-tauri/src/managed_agents/nest_skill.md @@ -87,14 +87,11 @@ Write commands are unaffected. `--format json` (default) returns full fields. ## Communication Patterns -**Mentions that notify:** Use `@Name` directly in message content — the CLI auto-resolves channel members by name and adds the required p-tags. No `--mention` flag exists or is needed. `nostr:npub1…` inline references are also auto-resolved to p-tags without needing a flag. +**Mentions that notify:** Keep readable `@Name` text in message content and, when intended pubkeys are known, pass the identities in the same send with repeatable `--mention `. Any explicit identity (`--mention` or `nostr:npub...`) permits unresolved or ambiguous `@Name` text as presentation-only; uniquely resolved member names still add recipients. Include a pubkey for every presentation-only name that should notify. The CLI reports the signed event's `mention_pubkeys`; no follow-up verification command is needed. Without explicit identities, names resolve against current channel members. An unresolved/ambiguous name or non-member target stops before publishing. Add membership separately only when authorized, then retry; sending never changes membership automatically. ```bash -# ✅ Correct — notification delivered automatically -buzz messages send --channel --content "@Alice check this" - -# Multiple mentions — same pattern -buzz messages send --channel --content "@Alice @Bob review please" +buzz messages send --channel \ + --content "@Alice check this" --mention ``` ## DM Management From 581baa625400f72316776ba726f533735acf22db Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 16:38:44 -0400 Subject: [PATCH 21/32] chore(ci): bump Linux AppImage build container to ubuntu:24.04 (#3602) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What and why The Buzz AppImage is built on `ubuntu:22.04`, which links WebKitGTK against FreeType 2.11.1. Because `libfreetype.so.6` is on the linuxdeploy community excludelist, the bundled WebKit loads the **host's** FreeType at runtime instead of the bundled one. FreeType 2.13.0 (released 2023-02-09) added `FT_Bool read_variable` to `FT_ColorStopIterator`, growing the struct from 16 to 20 bytes. Any host running FreeType ≥ 2.13 (Fedora 42+, Ubuntu 24.04+) has a struct-layout mismatch with the 22.04-compiled WebKit. The mismatched offsets corrupt color-stop index arithmetic inside Skia's COLRv1 renderer, producing the assertion abort in issues #2548 and #2982: ``` stl_vector.h:1123: Assertion '__n < this->size()' failed. ... colrv1_configure_skpaint(FT_Face, ...) ... ``` ## Fix Bump the build container to `ubuntu:24.04` (noble), which ships FreeType **2.13.2**. Noble's struct layout matches every crash-affected host. The ABI mismatch disappears and the crash is eliminated at root. WebKitGTK also advances from **2.50.4** (jammy backport) to **2.52.3** (noble backport). ## Glibc floor change | Build base | glibc floor | Oldest supported AppImage distro | |---|---|---| | ubuntu:22.04 (before) | 2.35 | Ubuntu 22.04 LTS, Debian 12 | | ubuntu:24.04 (after) | 2.39 | Ubuntu 24.04 LTS, Fedora 40+ | Ubuntu 22.04 LTS and Debian 12 users lose AppImage support. Both distributions continue to receive first-class `.deb` / `.rpm` packages, which use the system WebKit and are unaffected. The crash-affected users (Fedora 42/44, Ubuntu 24.04+) all have glibc ≥ 2.39. ## Changes - `.github/workflows/linux-canary.yml:24` — container pin updated to `ubuntu:24.04@sha256:4fbb8e6a…` - `.github/workflows/release.yml:479` — same container pin updated - `.github/workflows/release.yml:501` — comment version string updated from 22.04 to 24.04 `fix-appimage.sh` and `desktop/src-tauri/**` are untouched. The #3573 fontconfig stopgap remains active; retirement is a separate follow-on PR once this fix is verified on a shipped build. ## Sequencing `docs/linux-rendering-troubleshooting.md` (introduced in #3573) will receive a glibc-floor callout section once #3573 merges — adding it here would conflict with #3573's open branch. Context: #2548, #2982. Signed-off-by: Will Pfleger Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 --- .github/workflows/linux-canary.yml | 2 +- .github/workflows/release.yml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/linux-canary.yml b/.github/workflows/linux-canary.yml index 18d476e400..9806443378 100644 --- a/.github/workflows/linux-canary.yml +++ b/.github/workflows/linux-canary.yml @@ -21,7 +21,7 @@ jobs: name: Build Linux canary if: github.repository == 'block/buzz' runs-on: ubuntu-latest - container: ubuntu:22.04@sha256:0e0a0fc6d18feda9db1590da249ac93e8d5abfea8f4c3c0c849ce512b5ef8982 + container: ubuntu:24.04@sha256:4fbb8e6a8395de5a7550b33509421a2bafbc0aab6c06ba2cef9ebffbc7092d90 timeout-minutes: 60 permissions: contents: read diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c613924e57..b87e9c8c08 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -476,7 +476,7 @@ jobs: if: github.repository == 'block/buzz' runs-on: ubuntu-latest # Digest-pinned like the SHA-pinned actions below; Renovate keeps it fresh. - container: ubuntu:22.04@sha256:0e0a0fc6d18feda9db1590da249ac93e8d5abfea8f4c3c0c849ce512b5ef8982 + container: ubuntu:24.04@sha256:4fbb8e6a8395de5a7550b33509421a2bafbc0aab6c06ba2cef9ebffbc7092d90 needs: setup timeout-minutes: 60 permissions: @@ -498,7 +498,7 @@ jobs: env: DEBIAN_FRONTEND: noninteractive run: | - # Must run first: bare ubuntu:22.04 ships without curl, wget, git, or + # Must run first: bare ubuntu:24.04 ships without curl, wget, git, or # ca-certificates. activate-hermit bootstraps via curl+HTTPS (needs # both), and actions/checkout falls back to a REST tarball without git. # Running as root — no sudo needed. From 005b5b819a98ce85d4d80cd81b258fb6f9b8d51e Mon Sep 17 00:00:00 2001 From: Dave Grochowski Date: Wed, 29 Jul 2026 17:01:26 -0400 Subject: [PATCH 22/32] feat(tracing): correlate trace IDs in relay logs (#3608) ## Summary Correlates trace + span IDs with logs, allowing traces and logs to be bridged seamlessly ### Related issue none found ### Testing Unit tests Signed-off-by: David Grochowski Co-authored-by: Amp --- crates/buzz-relay/src/main.rs | 10 +- crates/buzz-relay/src/telemetry.rs | 362 ++++++++++++++++++++++++++++- 2 files changed, 368 insertions(+), 4 deletions(-) diff --git a/crates/buzz-relay/src/main.rs b/crates/buzz-relay/src/main.rs index 3ed820d3c5..9e6ca828e0 100644 --- a/crates/buzz-relay/src/main.rs +++ b/crates/buzz-relay/src/main.rs @@ -102,6 +102,7 @@ async fn main() -> anyhow::Result<()> { // spans under the correct service identity. let resource = telemetry::service_resource(); let tracer_init = telemetry::try_init_tracer(resource.clone()); + let otel_enabled = matches!(&tracer_init, telemetry::TracerInit::Enabled(_)); let otel_layer = match &tracer_init { telemetry::TracerInit::Enabled(p) => { use opentelemetry::trace::TracerProvider as _; @@ -109,12 +110,18 @@ async fn main() -> anyhow::Result<()> { } _ => None, }; + let trace_context_lookup = telemetry::TraceContextLookup::default(); + let trace_context_lookup_layer = otel_enabled.then(|| { + trace_context_lookup + .clone() + .with_filter(tracing_subscriber::filter::LevelFilter::OFF) + }); tracing_subscriber::registry() .with( fmt::layer() .json() - .flatten_event(true) + .event_format(trace_context_lookup.json_formatter(otel_enabled)) .with_filter(log_env_filter(std::env::var("RUST_LOG").ok().as_deref())), ) .with(otel_layer.map(|layer| { @@ -122,6 +129,7 @@ async fn main() -> anyhow::Result<()> { std::env::var("BUZZ_OTEL_FILTER").ok().as_deref(), )) })) + .with(trace_context_lookup_layer) .init(); // Log any exporter-build failure now that the subscriber is installed. diff --git a/crates/buzz-relay/src/telemetry.rs b/crates/buzz-relay/src/telemetry.rs index 11c6d03512..91bd92f0f3 100644 --- a/crates/buzz-relay/src/telemetry.rs +++ b/crates/buzz-relay/src/telemetry.rs @@ -23,9 +23,157 @@ //! - `OTEL_TRACES_SAMPLER` (default: `parentbased_always_on`) //! - `OTEL_TRACES_SAMPLER_ARG` +use std::{ + fmt, + sync::{Arc, OnceLock}, +}; + +use opentelemetry::trace::{SpanId, TraceContextExt as _, TraceId}; use opentelemetry_otlp::ExporterBuildError; use opentelemetry_sdk::{resource::EnvResourceDetector, trace::SdkTracerProvider, Resource}; -use tracing_subscriber::EnvFilter; +use tracing::{Event, Subscriber}; +use tracing_subscriber::{ + fmt::{ + format::{Format, FormatEvent, FormatFields, Json, Writer}, + FmtContext, + }, + registry::LookupSpan, + EnvFilter, Layer, +}; + +/// Captures the subscriber dispatch used to resolve tracing span IDs to their +/// OpenTelemetry contexts. +#[derive(Clone, Default)] +pub struct TraceContextLookup { + dispatch: Arc>, +} + +impl TraceContextLookup { + /// Build a JSON formatter backed by this subscriber dispatch lookup. + pub fn json_formatter(&self, enabled: bool) -> TraceContextJson { + TraceContextJson { + inner: tracing_subscriber::fmt::format().json().flatten_event(true), + enabled, + context_lookup: self.clone(), + } + } + + fn nearest_otel_context(&self, span_id: &tracing::span::Id) -> Option { + let dispatch = self.dispatch.get()?.upgrade()?; + let registry = dispatch.downcast_ref::()?; + + let context = registry.span(span_id)?.scope().find_map(|span| { + let context = tracing_opentelemetry::get_otel_context(&span.id(), &dispatch)?; + context.span().span_context().is_valid().then_some(context) + }); + context + } +} + +impl Layer for TraceContextLookup { + fn on_register_dispatch(&self, subscriber: &tracing::Dispatch) { + let _ = self.dispatch.set(subscriber.downgrade()); + } +} + +/// JSON event formatter that adds the active OpenTelemetry trace context. +/// +/// Datadog recognizes the OpenTelemetry-standard `trace_id` and `span_id` +/// fields when they are lowercase hexadecimal strings. Events outside a valid +/// OpenTelemetry span retain the standard `tracing-subscriber` JSON format. +pub struct TraceContextJson { + inner: Format, + enabled: bool, + context_lookup: TraceContextLookup, +} + +struct CorrelationWriter<'writer> { + inner: Writer<'writer>, + trace_id: TraceId, + span_id: SpanId, + injected: bool, +} + +impl fmt::Write for CorrelationWriter<'_> { + fn write_str(&mut self, value: &str) -> fmt::Result { + if self.injected { + return self.inner.write_str(value); + } + + let Some(object_start) = value.find('{') else { + return self.inner.write_str(value); + }; + self.inner.write_str(&value[..=object_start])?; + write!( + self.inner, + "\"trace_id\":\"{}\",\"span_id\":\"{}\",", + self.trace_id, self.span_id + )?; + self.injected = true; + self.inner.write_str(&value[object_start + 1..]) + } +} + +impl FormatEvent for TraceContextJson +where + S: Subscriber + for<'lookup> LookupSpan<'lookup>, + N: for<'writer> FormatFields<'writer> + 'static, +{ + fn format_event( + &self, + ctx: &FmtContext<'_, S, N>, + mut writer: Writer<'_>, + event: &Event<'_>, + ) -> fmt::Result { + if !self.enabled { + return self.inner.format_event(ctx, writer, event); + } + + let otel_context = match event.parent() { + Some(span_id) => self.context_lookup.nearest_otel_context(span_id), + None if event.is_contextual() => Some(opentelemetry::Context::current()), + None => None, + }; + let Some(otel_context) = otel_context else { + return self.inner.format_event(ctx, writer, event); + }; + let otel_span = otel_context.span(); + let span_context = otel_span.span_context(); + + if !span_context.is_valid() { + return self.inner.format_event(ctx, writer, event); + } + + let trace_id = span_context.trace_id(); + let span_id = span_context.span_id(); + + // Events may define fields with the correlation names themselves. In + // that uncommon case, overwrite them rather than emitting duplicate + // JSON keys. Preserve the allocation-free streaming path for ordinary + // events. + let fields = event.metadata().fields(); + if fields.field("trace_id").is_some() || fields.field("span_id").is_some() { + let mut json = String::new(); + self.inner + .format_event(ctx, Writer::new(&mut json), event)?; + let mut object: serde_json::Map = + serde_json::from_str(json.trim_end()).map_err(|_| fmt::Error)?; + object.insert("trace_id".into(), trace_id.to_string().into()); + object.insert("span_id".into(), span_id.to_string().into()); + writer.write_str(&serde_json::to_string(&object).map_err(|_| fmt::Error)?)?; + return writeln!(writer); + } + + let mut writer = CorrelationWriter { + inner: writer, + trace_id, + span_id, + injected: false, + }; + self.inner + .format_event(ctx, Writer::new(&mut writer), event) + } +} /// Build the filter for spans exported through OpenTelemetry. /// @@ -122,8 +270,13 @@ fn classify_exporter_result( #[cfg(test)] mod tests { use super::*; - use opentelemetry::KeyValue; - use std::sync::Mutex; + use opentelemetry::{trace::TracerProvider as _, KeyValue}; + use opentelemetry_sdk::trace::InMemorySpanExporter; + use std::{ + io, + sync::{Arc, Mutex}, + }; + use tracing_subscriber::prelude::*; // Env vars are process-global — serialize tests that mutate them to prevent // cross-test races when the suite runs with multiple threads. @@ -137,6 +290,209 @@ mod tests { .map(|(_, v)| v.to_string()) } + #[derive(Clone)] + struct CapturingWriter(Arc>>); + + impl io::Write for CapturingWriter { + fn write(&mut self, bytes: &[u8]) -> io::Result { + self.0.lock().unwrap().extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } + } + + #[test] + fn trace_context_json_correlates_nested_span_logs() { + let output = Arc::new(Mutex::new(Vec::new())); + let output_writer = Arc::clone(&output); + let exporter = InMemorySpanExporter::default(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let tracer = provider.tracer("trace-context-json-test"); + let context_lookup = TraceContextLookup::default(); + + let subscriber = tracing_subscriber::registry() + .with( + tracing_subscriber::fmt::layer() + .json() + .event_format(context_lookup.json_formatter(true)) + .with_writer(move || CapturingWriter(Arc::clone(&output_writer))) + .with_filter(tracing_subscriber::filter::filter_fn(|metadata| { + metadata.target() != "stdout_filtered" + })), + ) + .with( + tracing_opentelemetry::layer() + .with_tracer(tracer) + .with_filter(tracing_subscriber::filter::filter_fn(|metadata| { + !matches!(metadata.target(), "filtered" | "otel_event_filtered") + })), + ) + .with( + context_lookup + .clone() + .with_filter(tracing_subscriber::filter::LevelFilter::OFF), + ); + + tracing::subscriber::with_default(subscriber, || { + let explicit = tracing::info_span!("explicit"); + let root = tracing::info_span!("root"); + root.in_scope(|| { + tracing::info!(answer = 42, "root event"); + tracing::info!( + trace_id = "event-provided-trace", + span_id = "event-provided-span", + "colliding-fields event" + ); + tracing::info!(parent: &explicit, "explicit-parent event"); + tracing::info!(parent: None, "explicit-root event"); + let child = tracing::info_span!("child"); + child.in_scope(|| tracing::info!("child event")); + + let filtered_child = tracing::info_span!(target: "filtered", "filtered-child"); + filtered_child.in_scope(|| tracing::info!("filtered-child event")); + tracing::info!( + parent: &filtered_child, + "explicit-filtered-child event" + ); + + let stdout_filtered_child = + tracing::info_span!(target: "stdout_filtered", "stdout-filtered-child"); + stdout_filtered_child.in_scope(|| tracing::info!("stdout-filtered-child event")); + + tracing::info!(target: "otel_event_filtered", "otel-filtered event"); + }); + let filtered = tracing::info_span!(target: "filtered", "filtered"); + filtered.in_scope(|| tracing::info!("filtered-span event")); + tracing::info!("unscoped event"); + }); + + provider.force_flush().unwrap(); + let spans = exporter.get_finished_spans().unwrap(); + let root = spans.iter().find(|span| span.name == "root").unwrap(); + let explicit = spans.iter().find(|span| span.name == "explicit").unwrap(); + let child = spans.iter().find(|span| span.name == "child").unwrap(); + let stdout_filtered_child = spans + .iter() + .find(|span| span.name == "stdout-filtered-child") + .unwrap(); + + let bytes = output.lock().unwrap().clone(); + let output = String::from_utf8(bytes).unwrap(); + let lines: Vec<&str> = output.lines().collect(); + let logs: Vec = lines + .iter() + .map(|line| serde_json::from_str(line).unwrap()) + .collect(); + assert_eq!(logs.len(), 11); + + assert_eq!(logs[0]["message"], "root event"); + assert_eq!(logs[0]["answer"], 42); + assert_eq!( + logs[0]["trace_id"], + root.span_context.trace_id().to_string() + ); + assert_eq!(logs[0]["span_id"], root.span_context.span_id().to_string()); + assert_eq!(logs[0]["trace_id"].as_str().unwrap().len(), 32); + assert_eq!(logs[0]["span_id"].as_str().unwrap().len(), 16); + + assert_eq!(logs[1]["message"], "colliding-fields event"); + assert_eq!( + logs[1]["trace_id"], + root.span_context.trace_id().to_string() + ); + assert_eq!(logs[1]["span_id"], root.span_context.span_id().to_string()); + assert_eq!(lines[1].matches("\"trace_id\":").count(), 1); + assert_eq!(lines[1].matches("\"span_id\":").count(), 1); + + assert_eq!(logs[2]["message"], "explicit-parent event"); + assert_eq!( + logs[2]["trace_id"], + explicit.span_context.trace_id().to_string() + ); + assert_eq!( + logs[2]["span_id"], + explicit.span_context.span_id().to_string() + ); + + assert_eq!(logs[3]["message"], "explicit-root event"); + assert!(logs[3].get("trace_id").is_none()); + assert!(logs[3].get("span_id").is_none()); + + assert_eq!(logs[4]["message"], "child event"); + assert_eq!( + logs[4]["trace_id"], + child.span_context.trace_id().to_string() + ); + assert_eq!(logs[4]["span_id"], child.span_context.span_id().to_string()); + assert_eq!(logs[0]["trace_id"], logs[4]["trace_id"]); + + assert_eq!(logs[5]["message"], "filtered-child event"); + assert_eq!( + logs[5]["trace_id"], + root.span_context.trace_id().to_string() + ); + assert_eq!(logs[5]["span_id"], root.span_context.span_id().to_string()); + + assert_eq!(logs[6]["message"], "explicit-filtered-child event"); + assert_eq!( + logs[6]["trace_id"], + root.span_context.trace_id().to_string() + ); + assert_eq!(logs[6]["span_id"], root.span_context.span_id().to_string()); + + assert_eq!(logs[7]["message"], "stdout-filtered-child event"); + assert_eq!( + logs[7]["trace_id"], + stdout_filtered_child.span_context.trace_id().to_string() + ); + assert_eq!( + logs[7]["span_id"], + stdout_filtered_child.span_context.span_id().to_string() + ); + + assert_eq!(logs[8]["message"], "otel-filtered event"); + assert_eq!( + logs[8]["trace_id"], + root.span_context.trace_id().to_string() + ); + assert_eq!(logs[8]["span_id"], root.span_context.span_id().to_string()); + + assert_eq!(logs[9]["message"], "filtered-span event"); + assert!(logs[9].get("trace_id").is_none()); + assert!(logs[9].get("span_id").is_none()); + + assert_eq!(logs[10]["message"], "unscoped event"); + assert!(logs[10].get("trace_id").is_none()); + assert!(logs[10].get("span_id").is_none()); + } + + #[test] + fn trace_context_lookup_does_not_enable_callsites() { + let context_lookup = TraceContextLookup::default(); + let subscriber = tracing_subscriber::registry().with( + context_lookup + .clone() + .with_filter(tracing_subscriber::filter::LevelFilter::OFF), + ); + + tracing::subscriber::with_default(subscriber, || { + assert!(context_lookup + .dispatch + .get() + .and_then(tracing::dispatcher::WeakDispatch::upgrade) + .is_some()); + assert!(!tracing::enabled!( + target: "trace_context_lookup_filter_test", + tracing::Level::ERROR + )); + }); + } + #[test] fn test_service_resource_default_when_env_unset() { let _guard = ENV_LOCK.lock().unwrap(); From 5aeed7c7a2f89f9ae201624ffd431ca12de624f9 Mon Sep 17 00:00:00 2001 From: Xule Lin <43122877+linxule@users.noreply.github.com> Date: Wed, 29 Jul 2026 22:50:55 +0100 Subject: [PATCH 23/32] fix(desktop): discover bun-installed agent CLIs in ~/.bun/bin (#3343) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem `common_binary_paths()` probes mise shims, `~/.local/bin`, volta, asdf, and (further down `resolve_command_uncached`) nvm's default bin dir — but not bun's global bin directory, `~/.bun/bin`. bun's installer appends its bin dir to `~/.zshrc` / `~/.bashrc`, which are **interactive**-only. A login shell never sources them, so `find_via_login_shell()` can't recover the path either. That's the same failure mode already called out in this file for nvm: ```rust // Check nvm's default Node.js bin directory — nvm initializes via // ~/.zshrc (interactive) which is not loaded by a login shell, so // `node`, `npm`, and npm-global shims installed there are otherwise // invisible. ``` So for a GUI-launched desktop app, every rung of the resolution ladder misses a bun-installed CLI: 1. workspace dev dirs — no 2. `command_looks_like_path` — no, presets use bare names 3. Buzz-managed npm/node dirs — no 4. current process PATH — launchd's minimal PATH on a Finder launch 5. `find_via_login_shell` — `.zshrc` not sourced 6. `common_binary_paths()` — **`~/.bun/bin` absent** 7. nvm default bin — no This matters because bun is a common install route for the agent CLIs Buzz targets. Kimi Code in particular ships as an npm package (`@moonshot-ai/kimi-code`), so `bun add -g` puts it at `~/.bun/bin/kimi` — exactly where discovery doesn't look. ## Reproduction On macOS with `codex` and `kimi` installed via bun, launching Buzz from Finder: - Kimi Code shows **"CLI needed"** - both CLIs run fine in an interactive terminal Probing the way `find_via_login_shell` does, in a clean environment: ```console $ env -i HOME=$HOME /bin/zsh -l -c 'command -v -- codex; command -v -- kimi' (nothing) ``` Launching the app with the bun dir on PATH resolves both immediately: ```console $ env PATH="$HOME/.bun/bin:$PATH" /Applications/Buzz.app/Contents/MacOS/buzz-desktop ``` ## Change One entry appended to the home-relative list in `common_binary_paths()`. It goes **last** so it cannot shadow a directory that already resolves — the change can only add resolutions, never alter existing ones. ## Testing `cargo fmt --check` passes. I was not able to run the full `just ci` gate locally: `ring 0.17.14` fails to build in this environment against the macOS 26.2 SDK (`cc` error compiling `p256-nistz.c`), which is unrelated to this change. Relying on CI for the rest — the diff adds one `PathBuf` to an existing `Vec` and introduces no new API. ## Notes - Related to #3084, which adds `~/.kimi-code/bin` for the same class of GUI-launch discovery failure. That covers Kimi's standalone installer; this covers the bun/npm-global install route. They're complementary — I've left a note on that PR. - Only `~/.bun/bin` is added. bun's global packages live under `~/.bun/install/global/node_modules` but are symlinked into `~/.bun/bin`, so the single directory is sufficient. - Worth noting `~/.bun/bin` contains no `node`/`npm`/`npx`, so appending it can't shadow a system Node toolchain. Signed-off-by: Xule Lin <43122877+linxule@users.noreply.github.com> --- desktop/src-tauri/src/managed_agents/discovery.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/desktop/src-tauri/src/managed_agents/discovery.rs b/desktop/src-tauri/src/managed_agents/discovery.rs index eecbf4de3e..c8a85be34a 100644 --- a/desktop/src-tauri/src/managed_agents/discovery.rs +++ b/desktop/src-tauri/src/managed_agents/discovery.rs @@ -19,7 +19,6 @@ const CLAUDE_CODE_AVATAR_URL: &str = "https://anthropic.gallerycdn.vsassets.io/e const CODEX_AVATAR_URL: &str = "https://openai.gallerycdn.vsassets.io/extensions/openai/chatgpt/26.5313.41514/1773706730621/Microsoft.VisualStudio.Services.Icons.Default"; const BUZZ_AGENT_AVATAR_URL: &str = "https://raw.githubusercontent.com/block/buzz/refs/heads/main/crates/buzz-agent/buzz-agent.png"; - fn common_binary_paths() -> &'static [PathBuf] { static PATHS: OnceLock> = OnceLock::new(); PATHS.get_or_init(|| { @@ -41,6 +40,7 @@ fn common_binary_paths() -> &'static [PathBuf] { home.join(".local/bin"), home.join(".volta/bin"), home.join(".asdf/shims"), + home.join(".bun/bin"), ]); } // Windows well-known dirs for npm global shims and standalone installer targets. From b18e559ae2fbbb0a064f59afbdf1e2675fe86ea3 Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 17:54:33 -0400 Subject: [PATCH 24/32] docs: add Linux rendering troubleshooting guide (#3573) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Adds `docs/linux-rendering-troubleshooting.md` — the user-facing troubleshooting page for Linux rendering failures. ## What's in the doc **Crash: `colrv1_configure_skpaint` assertion abort (AppImage, Fedora 40+)** Root cause: the AppImage bundles WebKitGTK compiled against FreeType 2.11.1, but `libfreetype.so.6` is not bundled — WebKit loads the host's FreeType at runtime. FreeType 2.13.0 added a field to `FT_ColorStopIterator` (16 → 20 bytes); on hosts with FreeType ≥ 2.13 the struct-layout mismatch corrupts Skia's COLRv1 color-stop arithmetic, causing the assertion abort. Fix: upgrade to v0.5.2+ (build container bumped to `ubuntu:24.04` in [#3602](https://github.com/block/buzz/pull/3602)). Includes the glibc floor table (2.35 → 2.39) and `.deb`/`.rpm` guidance for Ubuntu 22.04 / Debian 12 users. A manual fontconfig workaround is preserved for users stuck on older AppImages. **Blank window / dmabuf renderer (NVIDIA, AppImage)** Covers the auto-fix shipped in v0.5.1 ([#3271](https://github.com/block/buzz/pull/3271)) and the `--safe-rendering` flag for cases where auto-detection misses. **AMD RDNA4 / transparent window ([#2643](https://github.com/block/buzz/issues/2643))** Documents the three-variable workaround verified by the reporter (`GDK_BACKEND=x11`, `WEBKIT_DISABLE_DMABUF_RENDERER=1`, `WEBKIT_SKIA_ENABLE_CPU_RENDERING=1`). Also includes a crash-log capture recipe and issue-filing checklist. Context: [#2548](https://github.com/block/buzz/issues/2548), [#2982](https://github.com/block/buzz/issues/2982), [#2643](https://github.com/block/buzz/issues/2643), [#2338](https://github.com/block/buzz/issues/2338). Signed-off-by: Will Pfleger Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 --- docs/linux-rendering-troubleshooting.md | 135 ++++++++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 docs/linux-rendering-troubleshooting.md diff --git a/docs/linux-rendering-troubleshooting.md b/docs/linux-rendering-troubleshooting.md new file mode 100644 index 0000000000..1e09ef1aca --- /dev/null +++ b/docs/linux-rendering-troubleshooting.md @@ -0,0 +1,135 @@ +# Linux Rendering Troubleshooting + +This guide covers the most common rendering failures on Linux and how to resolve them. It covers both the AppImage distribution and native package installs (`deb`, `rpm`). + +## Symptoms and fixes at a glance + +| Symptom | Likely cause | Fix | +|---------|-------------|-----| +| Blank or transparent window, then `SIGABRT` with `colrv1_configure_skpaint` in the output | COLRv1 color emoji font (AppImage only) | Upgrade to the latest AppImage (v0.5.2+) | +| Blank window on startup, no crash output | dmabuf renderer incompatibility (NVIDIA or AppImage) | `WEBKIT_DISABLE_DMABUF_RENDERER=1 ./Buzz.AppImage` or `--safe-rendering` | +| Blank window on any hardware, no crash output | Unknown GPU/driver combination | `--safe-rendering` flag (see below) | + +--- + +## Crash: `colrv1_configure_skpaint` assertion abort (AppImage) + +**Affected distributions:** Fedora 40+ and any distro shipping Google's Noto Color Emoji in COLRv1 format (`Noto-COLRv1.ttf`). Issues [#2548](https://github.com/block/buzz/issues/2548), [#2982](https://github.com/block/buzz/issues/2982). + +**Symptom:** Buzz starts, the window appears briefly (or stays blank), then the process aborts with output like: + +``` +././/include/c++/12/bits/stl_vector.h:1123: ... colrv1_configure_skpaint ...: +Assertion '__n < this->size()' failed. +``` + +**Root cause:** The AppImage bundles WebKitGTK compiled against FreeType 2.11.1 (Ubuntu 22.04's version), but `libfreetype.so.6` is not bundled — WebKit loads the host's FreeType at runtime instead. FreeType 2.13.0 (2023-02-09) added a field to `FT_ColorStopIterator`, growing the struct from 16 to 20 bytes. On Fedora 40+ hosts (FreeType ≥ 2.13), the struct-layout mismatch corrupts color-stop index arithmetic inside Skia's COLRv1 renderer, producing the assertion abort. + +**Fix:** Upgrade to the latest AppImage (v0.5.2+). The build container was bumped to `ubuntu:24.04` ([#3602](https://github.com/block/buzz/pull/3602)), which ships FreeType 2.13.2. The compiled layout now matches every crash-affected host (FreeType ≥ 2.13), eliminating the ABI mismatch. + +**AppImage glibc floor (v0.5.2+):** The `ubuntu:24.04` build raises the AppImage's minimum glibc requirement: + +| AppImage version | glibc floor | Oldest supported AppImage distro | +|---|---|---| +| v0.5.1 and earlier | 2.35 | Ubuntu 22.04 LTS, Debian 12 | +| v0.5.2+ | 2.39 | Ubuntu 24.04 LTS, Fedora 40+ | + +If you are on **Ubuntu 22.04 LTS or Debian 12**, upgrade to the latest **`.deb`/`.rpm`** package instead — native packages use the system WebKit and are unaffected by this change. + +**Workaround (before upgrading):** Add a fontconfig override that removes color-format fonts from Buzz's view: + +```bash +mkdir -p ~/.config/buzz-fontconfig +cat > ~/.config/buzz-fontconfig/fonts.conf <<'XML' + + + + /etc/fonts/fonts.conf + + + + true + + + + +XML +FONTCONFIG_FILE=~/.config/buzz-fontconfig/fonts.conf ./Buzz_*.AppImage +``` + +**Native packages (`deb`/`rpm`):** The COLRv1 crash ([#2548](https://github.com/block/buzz/issues/2548), [#2982](https://github.com/block/buzz/issues/2982)) is AppImage-only — native packages use the system WebKit, which has a consistent FreeType ABI, and are not affected. + +--- + +## Blank window on startup (no crash): dmabuf renderer + +**Affected hardware:** NVIDIA GPUs (proprietary and nouveau drivers) and AppImage installs on any GPU. Issue [#2338](https://github.com/block/buzz/issues/2338). + +**Symptom:** Buzz launches without any crash or assertion output, but the window is blank or invisible. The process is running (`ps aux | grep buzz`), but nothing renders. + +**Root cause:** WebKitGTK's dmabuf zero-copy buffer path is incompatible with some GPU/driver/compositor combinations. The WebKit child process silently fails to paint. + +**Fix (shipped automatically starting with the first release containing [#3271](https://github.com/block/buzz/pull/3271) (v0.5.1)):** Buzz sets `WEBKIT_DISABLE_DMABUF_RENDERER=1` automatically before WebKit initializes when it detects an NVIDIA GPU (`/sys/class/drm` vendor ID `0x10de`) or when running as an AppImage. This restores a slightly slower shared-memory rendering path that works universally. + +**If automatic detection doesn't help (`--safe-rendering`):** Pass `--safe-rendering` to force both `WEBKIT_DISABLE_DMABUF_RENDERER=1` and `WEBKIT_DISABLE_COMPOSITING_MODE=1` for that launch: + +```bash +./Buzz_*.AppImage --safe-rendering +# or for a native install: +buzz-desktop --safe-rendering +``` + +`--safe-rendering` is a per-launch flag — it is not remembered between runs. If it fixes your issue, you can make it permanent by setting the env vars yourself: + +```bash +# ~/.bashrc or ~/.profile +export WEBKIT_DISABLE_DMABUF_RENDERER=1 +``` + +**Conflict detection:** If you set a WebKit variable in your environment and also pass `--safe-rendering`, Buzz will refuse to start and print exactly which variable conflicts. Unset the conflicting variable or drop the flag. + +--- + +## AMD RDNA4 / transparent window + +**Affected hardware:** AMD RDNA4 GPUs (RX 9000 series) with the `radv` driver. Issue [#2643](https://github.com/block/buzz/issues/2643). + +**Symptom:** The Buzz window is transparent or renders with graphical corruption on AMD RDNA4 hardware. + +**Workaround (verified by reporter):** Set these three variables before launching Buzz: + +```bash +export GDK_BACKEND=x11 +export WEBKIT_DISABLE_DMABUF_RENDERER=1 +export WEBKIT_SKIA_ENABLE_CPU_RENDERING=1 +./Buzz_*.AppImage +# or for native: +buzz-desktop +``` + +- `WEBKIT_SKIA_ENABLE_CPU_RENDERING=1` forces Skia to use CPU rendering, bypassing the RDNA4 Skia/radv paint failure. +- `GDK_BACKEND=x11` avoids the blank window that appears when running under a Plasma-Wayland compositor. +- `WEBKIT_DISABLE_DMABUF_RENDERER=1` prevents post-first-paint transparency from the dmabuf renderer. + +A dedicated fix for RDNA4 detection is being tracked in [#2643](https://github.com/block/buzz/issues/2643). + +--- + +## Diagnosing an unrecognised crash + +If none of the above match your situation: + +1. Run Buzz from a terminal and capture the output: + ```bash + ./Buzz_*.AppImage 2>&1 | tee buzz-crash.log + ``` + +2. Check for a core dump: + ```bash + coredumpctl list | tail + coredumpctl info + ``` + +3. Try `--safe-rendering` first — if it resolves the issue, it's a WebKit rendering incompatibility and the crash log will help narrow down which driver is involved. + +4. File a [new issue](https://github.com/block/buzz/issues/new) with your distro, GPU, driver version, and the terminal output. From 3e48f1b2365d326ee1c9582448d86a99b44ecd5d Mon Sep 17 00:00:00 2001 From: Wes Date: Wed, 29 Jul 2026 15:57:35 -0600 Subject: [PATCH 25/32] chore(release): release Buzz Desktop version 0.5.2 (#3624) ## Buzz Desktop release v0.5.2 ### Changes since v0.5.1: - feat(cli): mirror Desktop mention delivery ([#3330](https://github.com/block/buzz/pull/3330)) ([`7adc46268`](https://github.com/block/buzz/commit/7adc46268d5e93f0b1d4dc8e700af22815dcac1b)) - fix(desktop): deduplicate relay outage notification ([#3579](https://github.com/block/buzz/pull/3579)) ([`66e705492`](https://github.com/block/buzz/commit/66e7054928cc29395f828467c3e8c81b7408dd29)) - fix(desktop): reconcile thread arrivals at bottom ([#3585](https://github.com/block/buzz/pull/3585)) ([`b42a8d447`](https://github.com/block/buzz/commit/b42a8d447e3a2b85b2313dc4fdd123731fd8bba3)) - Improve emoji autocomplete matching ([#3571](https://github.com/block/buzz/pull/3571)) ([`259de6afb`](https://github.com/block/buzz/commit/259de6afbe0cc0d106e57ebdb2323064990e4122)) - Fix shared agent avatar import profiles ([#3578](https://github.com/block/buzz/pull/3578)) ([`324bd6b46`](https://github.com/block/buzz/commit/324bd6b464de5751e12abbd155376046ce3d2afc)) - Fix inline raster avatars in agent catalog ([#3581](https://github.com/block/buzz/pull/3581)) ([`7e9b77f72`](https://github.com/block/buzz/commit/7e9b77f72d82e019a99f074f1c9829be30c57ae1)) - feat(agent): make Gemini and MLflow-route models usable through databricks_v2 ([#3569](https://github.com/block/buzz/pull/3569)) ([`4a1ebf25c`](https://github.com/block/buzz/commit/4a1ebf25c782fc6a68f0a69e6f866f793a259a1f)) **To release:** merge this PR. The tag and build will happen automatically. Signed-off-by: Wes --- CHANGELOG.md | 11 +++++++++++ desktop/package.json | 2 +- desktop/src-tauri/Cargo.lock | 2 +- desktop/src-tauri/Cargo.toml | 2 +- desktop/src-tauri/tauri.conf.json | 2 +- 5 files changed, 15 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 956faa1ed3..d83087fc26 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## v0.5.2 + +- feat(cli): mirror Desktop mention delivery ([#3330](https://github.com/block/buzz/pull/3330)) ([`7adc46268`](https://github.com/block/buzz/commit/7adc46268d5e93f0b1d4dc8e700af22815dcac1b)) +- fix(desktop): deduplicate relay outage notification ([#3579](https://github.com/block/buzz/pull/3579)) ([`66e705492`](https://github.com/block/buzz/commit/66e7054928cc29395f828467c3e8c81b7408dd29)) +- fix(desktop): reconcile thread arrivals at bottom ([#3585](https://github.com/block/buzz/pull/3585)) ([`b42a8d447`](https://github.com/block/buzz/commit/b42a8d447e3a2b85b2313dc4fdd123731fd8bba3)) +- Improve emoji autocomplete matching ([#3571](https://github.com/block/buzz/pull/3571)) ([`259de6afb`](https://github.com/block/buzz/commit/259de6afbe0cc0d106e57ebdb2323064990e4122)) +- Fix shared agent avatar import profiles ([#3578](https://github.com/block/buzz/pull/3578)) ([`324bd6b46`](https://github.com/block/buzz/commit/324bd6b464de5751e12abbd155376046ce3d2afc)) +- Fix inline raster avatars in agent catalog ([#3581](https://github.com/block/buzz/pull/3581)) ([`7e9b77f72`](https://github.com/block/buzz/commit/7e9b77f72d82e019a99f074f1c9829be30c57ae1)) +- feat(agent): make Gemini and MLflow-route models usable through databricks_v2 ([#3569](https://github.com/block/buzz/pull/3569)) ([`4a1ebf25c`](https://github.com/block/buzz/commit/4a1ebf25c782fc6a68f0a69e6f866f793a259a1f)) + + ## v0.5.1 - perf(desktop): move observer-feed archive and decrypt commands off main thread ([#3415](https://github.com/block/buzz/pull/3415)) ([`294c8c821`](https://github.com/block/buzz/commit/294c8c821de51442a8c384c0bdb66b1a10224ca0)) diff --git a/desktop/package.json b/desktop/package.json index 7943b949b9..2226a0cb12 100644 --- a/desktop/package.json +++ b/desktop/package.json @@ -1,7 +1,7 @@ { "name": "buzz", "private": true, - "version": "0.5.1", + "version": "0.5.2", "type": "module", "scripts": { "dev": "vite", diff --git a/desktop/src-tauri/Cargo.lock b/desktop/src-tauri/Cargo.lock index d4f7a4a2d4..bf84cd0d33 100644 --- a/desktop/src-tauri/Cargo.lock +++ b/desktop/src-tauri/Cargo.lock @@ -1010,7 +1010,7 @@ dependencies = [ [[package]] name = "buzz-desktop" -version = "0.5.1" +version = "0.5.2" dependencies = [ "anyhow", "arboard", diff --git a/desktop/src-tauri/Cargo.toml b/desktop/src-tauri/Cargo.toml index 8bb643fea3..7606e48ac6 100644 --- a/desktop/src-tauri/Cargo.toml +++ b/desktop/src-tauri/Cargo.toml @@ -2,7 +2,7 @@ [package] name = "buzz-desktop" -version = "0.5.1" +version = "0.5.2" description = "Buzz desktop app" authors = ["you"] edition = "2021" diff --git a/desktop/src-tauri/tauri.conf.json b/desktop/src-tauri/tauri.conf.json index 85ad5c0b2d..2eba7815b2 100644 --- a/desktop/src-tauri/tauri.conf.json +++ b/desktop/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "Buzz", - "version": "0.5.1", + "version": "0.5.2", "identifier": "xyz.block.buzz.app", "build": { "beforeDevCommand": { From f95fdc1a102e17c6718a44323d9a2feaed702db7 Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 18:15:30 -0400 Subject: [PATCH 26/32] feat(agent,acp): wire provider total_tokens through NIP-AM publish chain (#3593) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Wires genuine provider-reported `total_tokens` through the full buzz-agent → buzz-acp publish chain so kind-44200 events carry real per-turn and cumulative totals for OpenAI-backed models, while preserving all existing behaviour for Anthropic and external harnesses (goose, claude-code). ## Why Live prod data showed 0 of 1,934 archived reports carry `totalTokens`. Both hardcoded `total_tokens: None` in `pool.rs` and the absent field in `buzz-agent`'s parser are root causes. This is the backend half of a two-track fix; the display-fallback half lands in [#2035](https://github.com/block/buzz/pull/2035). ## Changes **`crates/buzz-agent/src/types.rs`** - Added `total_tokens: Option` to `LlmResponse` with an explicit doc comment that NIP-AM forbids deriving it. - Added `TurnTotalState` enum (`Unseen | Exact(u64) | Unknown`) with `fold()` and `exact_value()` — the tri-state accumulator that distinguishes not-yet-observed from permanently poisoned. **`crates/buzz-agent/src/llm.rs`** - `parse_responses` and `parse_openai`: read `usage.total_tokens` from OpenAI Chat Completions (including Databricks routes) and the Responses API via `sum_usage`. - Anthropic: explicit `total_tokens: None` — no genuine total available; NIP-AM forbids summing categories. **`crates/buzz-agent/src/agent.rs`** - Added `turn_total_state: &'a mut TurnTotalState` to `RunCtx`. - Fold `response.total_tokens` into the accumulator after each usage-bearing response; non-usage-bearing responses (keepalive/stream frames) do not poison. **`crates/buzz-agent/src/lib.rs`** - Added `accumulated_total_state: TurnTotalState` to `Session` (default `Unseen`). - Per-turn state passed to `RunCtx`, folded into session cumulative after each turn. - Emits `accumulatedTotalTokens` in `usage_update` only when cumulative is `Exact(n)`. **`crates/buzz-acp/src/usage.rs`** - Added `accumulated_total_tokens: Option` (serde default) to `UsageUpdatePayload` — optional for goose compat. - Added `last_total: Option` to `SessionState`. - Added `turn_total_tokens` and `cumulative_total_tokens` to `TurnUsage` (field-local — never affect `delta_reliable`). - Derive turn-total delta only when prev and current are both `Some` and monotonic; absence, decrease, or no baseline leaves only the total delta null without touching input/output reliability. **`crates/buzz-acp/src/pool.rs`** - Replaced both hardcoded `total_tokens: None` in `publish_agent_turn_metric` with `usage.turn_total_tokens` and `usage.cumulative_total_tokens`. ## Tests 20 new tests across the four touched files: | File | Tests | |------|-------| | `types.rs` | `TurnTotalState` fold, accumulation, exact_value, default (7 tests) | | `llm.rs` | Chat present/absent, Responses present/absent, Anthropic always-None (5 tests) | | `usage.rs` | First turn no baseline, second-turn delta, cumulative decrease (field-local), current absent, goose-shaped deserialization, baseline absent (6 tests) | | `pool.rs` | Exact turn+cumulative mapping, null totals never derived (2 tests) | `cargo test -p buzz-acp -p buzz-agent` — all passing, 0 failures. ## Scope Boundary: `crates/buzz-agent/**` + `crates/buzz-acp/**` only. Desktop unchanged. `costUsd` explicitly out of scope. Related: [#2035](https://github.com/block/buzz/pull/2035) --------- Signed-off-by: Will Pfleger Co-authored-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz> --- crates/buzz-acp/src/pool.rs | 185 +++++++++++++++++++--- crates/buzz-acp/src/usage.rs | 203 ++++++++++++++++++++++++ crates/buzz-agent/src/agent.rs | 28 +++- crates/buzz-agent/src/lib.rs | 68 +++++--- crates/buzz-agent/src/llm.rs | 90 +++++++++++ crates/buzz-agent/src/types.rs | 230 ++++++++++++++++++++++++++++ crates/buzz-agent/tests/fake_llm.rs | 181 ++++++++++++++++++++++ 7 files changed, 940 insertions(+), 45 deletions(-) diff --git a/crates/buzz-acp/src/pool.rs b/crates/buzz-acp/src/pool.rs index 038f8a714c..d1e005cbcc 100644 --- a/crates/buzz-acp/src/pool.rs +++ b/crates/buzz-acp/src/pool.rs @@ -3409,33 +3409,35 @@ fn acp_stop_to_core(r: &StopReason) -> buzz_core::agent_turn_metric::StopReason } } -/// Best-effort: build and publish a `kind:44200` NIP-AM agent turn metric event. +/// Build the `(turn, cumulative)` `TokenCounts` pair for a NIP-AM kind-44200 +/// payload from a completed `TurnUsage`. /// -/// Does nothing when `usage` is `None` (goose emitted no usage notification -/// for this turn) or when `owner_pubkey` is unconfigured (no NIP-AO identity). -/// Errors are logged at WARN and never surface to the caller — metric -/// publishing must never fail a turn. -async fn publish_agent_turn_metric( - ctx: &PromptContext, - usage: Option, - channel_id: Option, - session_id: &str, - turn_id: &str, - stop_reason: Option, +/// Extracted as a pure function so the mapping logic can be tested independently +/// of relay/crypto infrastructure. `publish_agent_turn_metric` is the only +/// production caller. +/// +/// - `turn` is `None` when `delta_reliable` is false; otherwise it carries the +/// per-turn i/o/total/cost deltas for this turn. +/// - `cumulative` always carries the session-aggregate i/o/cost totals. +/// `total_tokens` is `Some` only when the session accumulated a genuine +/// provider-reported total on every turn — never derived from i/o sums +/// (NIP-AM MUST NOT). +pub(crate) fn build_turn_metric_counts( + usage: &crate::usage::TurnUsage, +) -> ( + Option, + Option, ) { - use buzz_core::agent_turn_metric::{AgentTurnMetricPayload, TokenCounts}; - use nostr::{EventBuilder, Kind, Tag}; - - let (usage, owner_pk) = match (usage, ctx.agent_owner_pubkey.as_ref()) { - (Some(u), Some(pk)) => (u, pk), - _ => return, - }; + use buzz_core::agent_turn_metric::TokenCounts; let turn_counts = if usage.delta_reliable { Some(TokenCounts { input_tokens: usage.turn_input_tokens, output_tokens: usage.turn_output_tokens, - total_tokens: None, + // Field-local: present only when both the previous and current + // cumulative totals were available and monotonic. Never derived + // from input+output. + total_tokens: usage.turn_total_tokens, cost_usd: usage.turn_cost_usd, cache_read_tokens: None, cache_write_tokens: None, @@ -3450,11 +3452,40 @@ async fn publish_agent_turn_metric( let cumulative_counts = Some(TokenCounts { input_tokens: Some(usage.cumulative_input_tokens), output_tokens: Some(usage.cumulative_output_tokens), - total_tokens: None, + // Present when every turn in the session reported a genuine provider + // total. None when the session has never emitted one or any turn lacked + // one. Never derived from input+output (NIP-AM MUST NOT). + total_tokens: usage.cumulative_total_tokens, cost_usd: usage.cumulative_cost_usd, cache_read_tokens: None, cache_write_tokens: None, }); + (turn_counts, cumulative_counts) +} + +/// Best-effort: build and publish a `kind:44200` NIP-AM agent turn metric event. +/// +/// Does nothing when `usage` is `None` (goose emitted no usage notification +/// for this turn) or when `owner_pubkey` is unconfigured (no NIP-AO identity). +/// Errors are logged at WARN and never surface to the caller — metric +/// publishing must never fail a turn. +async fn publish_agent_turn_metric( + ctx: &PromptContext, + usage: Option, + channel_id: Option, + session_id: &str, + turn_id: &str, + stop_reason: Option, +) { + use buzz_core::agent_turn_metric::AgentTurnMetricPayload; + use nostr::{EventBuilder, Kind, Tag}; + + let (usage, owner_pk) = match (usage, ctx.agent_owner_pubkey.as_ref()) { + (Some(u), Some(pk)) => (u, pk), + _ => return, + }; + + let (turn_counts, cumulative_counts) = build_turn_metric_counts(&usage); let timestamp = chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Millis, true); let payload = AgentTurnMetricPayload { harness: ctx.harness_name.clone(), @@ -5238,9 +5269,11 @@ mod tests { delta_reliable: true, turn_input_tokens: Some(100), turn_output_tokens: Some(50), + turn_total_tokens: None, turn_cost_usd: None, cumulative_input_tokens: 100, cumulative_output_tokens: 50, + cumulative_total_tokens: None, cumulative_cost_usd: None, model: None, }; @@ -5270,9 +5303,11 @@ mod tests { delta_reliable: true, turn_input_tokens: Some(200), turn_output_tokens: Some(80), + turn_total_tokens: None, turn_cost_usd: Some(0.001), cumulative_input_tokens: 200, cumulative_output_tokens: 80, + cumulative_total_tokens: None, cumulative_cost_usd: Some(0.001), model: None, }; @@ -5303,9 +5338,11 @@ mod tests { delta_reliable: true, turn_input_tokens: Some(50), turn_output_tokens: Some(20), + turn_total_tokens: None, turn_cost_usd: None, cumulative_input_tokens: 150, cumulative_output_tokens: 70, + cumulative_total_tokens: None, cumulative_cost_usd: None, model: None, }; @@ -5336,9 +5373,11 @@ mod tests { delta_reliable: false, // first turn from buzz-agent turn_input_tokens: None, turn_output_tokens: None, + turn_total_tokens: None, turn_cost_usd: None, cumulative_input_tokens: 400, cumulative_output_tokens: 100, + cumulative_total_tokens: None, cumulative_cost_usd: None, model: None, }; @@ -5354,6 +5393,110 @@ mod tests { .await; } + /// `build_turn_metric_counts` maps exact turn and cumulative totals from + /// `TurnUsage` to the corresponding `TokenCounts.total_tokens` fields. + /// Reverting the production fields at the call site to `None` would break + /// this test; the test constrains the real code path. + #[test] + fn test_build_turn_metric_counts_exact_totals_map_through() { + let usage = crate::usage::TurnUsage { + session_id: "sess-total".to_string(), + turn_seq: 2, + delta_reliable: true, + turn_input_tokens: Some(100), + turn_output_tokens: Some(30), + turn_total_tokens: Some(130), // genuine per-turn total + turn_cost_usd: None, + cumulative_input_tokens: 500, + cumulative_output_tokens: 120, + cumulative_total_tokens: Some(620), // genuine cumulative total + cumulative_cost_usd: None, + model: None, + }; + + let (turn, cumulative) = crate::pool::build_turn_metric_counts(&usage); + + // Serialise to JSON — this is what ultimately goes on the wire. + let turn_json = serde_json::to_value(turn.as_ref().expect("turn counts present")).unwrap(); + let cum_json = + serde_json::to_value(cumulative.as_ref().expect("cumulative counts present")).unwrap(); + + // Per-turn total must be the genuine provider-reported value. + assert_eq!( + turn_json["totalTokens"], + serde_json::json!(130), + "per-turn total must map to TokenCounts.totalTokens in wire JSON" + ); + assert_eq!(turn_json["inputTokens"], serde_json::json!(100)); + assert_eq!(turn_json["outputTokens"], serde_json::json!(30)); + + // Cumulative total must be the genuine session total. + assert_eq!( + cum_json["totalTokens"], + serde_json::json!(620), + "cumulative total must map to TokenCounts.totalTokens in wire JSON" + ); + assert_eq!(cum_json["inputTokens"], serde_json::json!(500)); + assert_eq!(cum_json["outputTokens"], serde_json::json!(120)); + } + + /// When totals are absent, `build_turn_metric_counts` must produce null + /// `total_tokens` — never a derived input+output sum (NIP-AM MUST NOT). + /// Reverting the production fields to hardcoded `None` would leave this test + /// passing but input/output would disagree, making the null-path detectable. + #[test] + fn test_build_turn_metric_counts_null_totals_never_derived() { + let usage = crate::usage::TurnUsage { + session_id: "sess-nototal".to_string(), + turn_seq: 1, + delta_reliable: true, + turn_input_tokens: Some(200), + turn_output_tokens: Some(60), + turn_total_tokens: None, // provider did not supply a total + turn_cost_usd: None, + cumulative_input_tokens: 200, + cumulative_output_tokens: 60, + cumulative_total_tokens: None, // session has no total + cumulative_cost_usd: None, + model: None, + }; + + let (turn, cumulative) = crate::pool::build_turn_metric_counts(&usage); + + let turn_json = serde_json::to_value(turn.as_ref().expect("turn counts present")).unwrap(); + let cum_json = + serde_json::to_value(cumulative.as_ref().expect("cumulative counts present")).unwrap(); + + // total_tokens must be null in the wire JSON. + assert!( + turn_json["totalTokens"].is_null(), + "absent turn total must serialize as null — not derived from in+out" + ); + assert!( + cum_json["totalTokens"].is_null(), + "absent cumulative total must serialize as null — not derived from in+out" + ); + + // Input/output must still carry their real values. + assert_eq!( + turn_json["inputTokens"], + serde_json::json!(200), + "inputTokens must be present even when total is absent" + ); + assert_eq!( + turn_json["outputTokens"], + serde_json::json!(60), + "outputTokens must be present even when total is absent" + ); + + // The null total must not equal the input+output sum — it must be genuinely null. + let derived_sum = serde_json::json!(200u64 + 60u64); + assert_ne!( + turn_json["totalTokens"], derived_sum, + "total_tokens must never equal input+output when provider omitted it" + ); + } + fn make_prompt_context_no_owner() -> PromptContext { let agent_keys = nostr::Keys::generate(); make_prompt_context_impl(&agent_keys, None) diff --git a/crates/buzz-acp/src/usage.rs b/crates/buzz-acp/src/usage.rs index 8cca9c96f8..1629eee935 100644 --- a/crates/buzz-acp/src/usage.rs +++ b/crates/buzz-acp/src/usage.rs @@ -92,6 +92,14 @@ pub(crate) struct UsageUpdatePayload { #[serde(default)] pub accumulated_cached_input_tokens: u64, pub accumulated_cost: Option, + /// Session-cumulative genuine provider total tokens. Optional — only + /// emitted by buzz-agent when every turn in the session so far supplied a + /// provider-reported total. Absent for goose (field ignore-if-absent for + /// backward compat), for Anthropic-backed turns, and for sessions where any + /// turn lacked a provider total. NIP-AM forbids deriving this by summing + /// categories, so the UI must approximate when this field is absent. + #[serde(default)] + pub accumulated_total_tokens: Option, /// Effective model id for this turn. Optional — goose payloads that /// predate this field deserialize cleanly as `None`. #[serde(default)] @@ -113,6 +121,10 @@ struct SessionState { last_output: u64, /// Cumulative cost at the end of the LAST PUBLISHED turn. last_cost: Option, + /// Cumulative total tokens at the end of the LAST PUBLISHED turn. + /// `None` when the session has never emitted a provider total (Unseen) or + /// when any prior turn lacked one (poisoned). + last_total: Option, } /// Per-turn usage record exposed to `TurnCompletionGuard` for NIP-AM publishing. @@ -131,6 +143,11 @@ pub struct TurnUsage { pub turn_input_tokens: Option, /// Per-turn output token delta; `None` when unreliable. pub turn_output_tokens: Option, + /// Per-turn total token delta; `None` when the cumulative total is + /// unavailable (no baseline, non-monotonic, or either snapshot was absent). + /// Field-local: a missing total never flips `delta_reliable` or invalidates + /// `turn_input_tokens`/`turn_output_tokens`. + pub turn_total_tokens: Option, /// Per-turn cost delta (`current − previous`); `None` when unreliable or /// either snapshot is missing. pub turn_cost_usd: Option, @@ -138,6 +155,9 @@ pub struct TurnUsage { pub cumulative_input_tokens: u64, /// Session-cumulative output tokens as reported by goose at end of turn. pub cumulative_output_tokens: u64, + /// Session-cumulative genuine provider total tokens as reported by buzz-agent; + /// `None` when the session has never emitted one or any turn lacked one. + pub cumulative_total_tokens: Option, /// Session-cumulative estimated cost in USD; `None` if goose did not report it. pub cumulative_cost_usd: Option, /// Effective model id for this turn (maps to NIP-AM `model`). `None` if the @@ -218,6 +238,7 @@ impl UsageTracker { let current_input = payload.accumulated_input_tokens; let current_output = payload.accumulated_output_tokens; let current_cost = payload.accumulated_cost; + let current_total = payload.accumulated_total_tokens; // Determine whether this session is currently in-flight so we know // whether to set `pending`. We compute the delta regardless so that @@ -262,6 +283,17 @@ impl UsageTracker { } }; + // Total-token delta: field-local — never affects `delta_reliable` or + // the input/output deltas. Null when: no baseline exists, either + // snapshot is absent, or cumulative total decreased. + let turn_total = match self.sessions.get(session_id) { + Some(prev) => match (current_total, prev.last_total) { + (Some(cur), Some(p)) if cur >= p => Some(cur - p), + _ => None, // no baseline, absent on either side, or decrease + }, + None => None, // no baseline yet + }; + if is_in_flight { // In-flight-match: update pending with the latest cumulative values. // Baseline is NOT advanced here — it advances only on take(). @@ -271,9 +303,11 @@ impl UsageTracker { delta_reliable, turn_input_tokens: turn_input, turn_output_tokens: turn_output, + turn_total_tokens: turn_total, turn_cost_usd: turn_cost, cumulative_input_tokens: current_input, cumulative_output_tokens: current_output, + cumulative_total_tokens: current_total, cumulative_cost_usd: current_cost, model: payload.model.clone(), }); @@ -292,6 +326,7 @@ impl UsageTracker { last_input: current_input, last_output: current_output, last_cost: current_cost, + last_total: current_total, }, ); } @@ -319,6 +354,7 @@ impl UsageTracker { last_input: record.cumulative_input_tokens, last_output: record.cumulative_output_tokens, last_cost: record.cumulative_cost_usd, + last_total: record.cumulative_total_tokens, }, ); Some(record) @@ -368,6 +404,7 @@ mod tests { accumulated_output_tokens: output, accumulated_cached_input_tokens: 0, accumulated_cost: cost, + accumulated_total_tokens: None, model: None, } } @@ -380,6 +417,7 @@ mod tests { accumulated_output_tokens: output, accumulated_cached_input_tokens: 0, accumulated_cost: cost, + accumulated_total_tokens: None, model: None, } } @@ -877,6 +915,7 @@ mod tests { accumulated_output_tokens: output, accumulated_cached_input_tokens: 0, accumulated_cost: cost, + accumulated_total_tokens: None, model: model.map(str::to_string), } } @@ -929,4 +968,168 @@ mod tests { "TurnUsage.model must be None when payload omits the field" ); } + + // ── accumulatedTotalTokens: field-local delta, session poisoning ─────── + + fn payload_with_total(input: u64, output: u64, total: Option) -> UsageUpdatePayload { + UsageUpdatePayload { + used: input + output, + context_limit: 200_000, + accumulated_input_tokens: input, + accumulated_output_tokens: output, + accumulated_cached_input_tokens: 0, + accumulated_cost: None, + accumulated_total_tokens: total, + model: None, + } + } + + #[test] + fn first_update_without_baseline_turn_total_is_none() { + // No baseline exists → turn total null, but delta_reliable/input/output + // follow the normal first-turn rule (delta_reliable = false). + let mut tracker = UsageTracker::default(); + tracker.begin_turn("sess-t1"); + tracker.record("sess-t1", &payload_with_total(100, 20, Some(120))); + let usage = tracker.take().expect("pending"); + + assert!(!usage.delta_reliable, "first turn: delta unreliable"); + assert!( + usage.turn_total_tokens.is_none(), + "no baseline → turn total must be None" + ); + assert_eq!( + usage.cumulative_total_tokens, + Some(120), + "cumulative total passes through even on first turn" + ); + } + + #[test] + fn second_turn_with_totals_produces_turn_delta() { + let mut tracker = UsageTracker::default(); + // Turn 1 — establish baseline. + tracker.begin_turn("sess-t2"); + tracker.record("sess-t2", &payload_with_total(100, 20, Some(120))); + let _ = tracker.take(); + + // Turn 2 — delta is computable. + tracker.begin_turn("sess-t2"); + tracker.record("sess-t2", &payload_with_total(200, 50, Some(250))); + let usage = tracker.take().expect("pending"); + + assert!(usage.delta_reliable); + assert_eq!(usage.turn_total_tokens, Some(130)); // 250 - 120 + assert_eq!(usage.cumulative_total_tokens, Some(250)); + } + + #[test] + fn cumulative_total_decrease_leaves_turn_total_null_without_affecting_reliability() { + // Cumulative total decreases (e.g. counter reset) → turn total null, + // but delta_reliable and input/output are NOT affected (field-local). + let mut tracker = UsageTracker::default(); + tracker.begin_turn("sess-t3"); + tracker.record("sess-t3", &payload_with_total(500, 100, Some(600))); + let _ = tracker.take(); + + tracker.begin_turn("sess-t3"); + // Cumulative total decreased: 600 → 50. + tracker.record("sess-t3", &payload_with_total(600, 150, Some(50))); + let usage = tracker.take().expect("pending"); + + assert!( + usage.delta_reliable, + "input/output decrease would flip reliability; total decrease must not" + ); + assert_eq!(usage.turn_input_tokens, Some(100)); + assert_eq!(usage.turn_output_tokens, Some(50)); + assert!( + usage.turn_total_tokens.is_none(), + "cumulative total decrease → turn total null (field-local)" + ); + assert_eq!( + usage.cumulative_total_tokens, + Some(50), + "cumulative total from payload still passes through" + ); + } + + #[test] + fn cumulative_total_absent_on_current_turn_leaves_turn_total_null() { + // Goose-shaped payload: no accumulatedTotalTokens field at all. + let mut tracker = UsageTracker::default(); + tracker.begin_turn("sess-t4"); + tracker.record("sess-t4", &payload_with_total(100, 20, Some(120))); + let _ = tracker.take(); + + // Second turn: goose omits the total field entirely. + tracker.begin_turn("sess-t4"); + tracker.record("sess-t4", &payload_with_total(200, 50, None)); + let usage = tracker.take().expect("pending"); + + assert!(usage.delta_reliable, "input/output delta unaffected"); + assert_eq!(usage.turn_input_tokens, Some(100)); + assert_eq!(usage.turn_output_tokens, Some(30)); + assert!( + usage.turn_total_tokens.is_none(), + "absent field → null turn total" + ); + assert!( + usage.cumulative_total_tokens.is_none(), + "absent cumulative total passes through as None" + ); + } + + #[test] + fn goose_shaped_payload_without_accumulated_total_deserializes_correctly() { + // goose payloads lack accumulatedTotalTokens; the field must default + // to None without a deserialization error (ignore-if-absent contract). + let json = r#"{ + "sessionUpdate": "usage_update", + "accumulatedInputTokens": 1000, + "accumulatedOutputTokens": 200, + "accumulatedCost": 0.01 + }"#; + let variant: GooseSessionUpdateVariant = + serde_json::from_str(json).expect("must deserialize without accumulatedTotalTokens"); + let payload = match variant { + GooseSessionUpdateVariant::UsageUpdate(p) => p, + _ => panic!("expected UsageUpdate"), + }; + assert!( + payload.accumulated_total_tokens.is_none(), + "absent accumulatedTotalTokens must default to None" + ); + + // And it must flow through the tracker correctly. + let mut tracker = UsageTracker::default(); + tracker.begin_turn("sess-goose-nototal"); + tracker.record("sess-goose-nototal", &payload); + let usage = tracker.take().expect("pending"); + assert!( + usage.cumulative_total_tokens.is_none(), + "goose-shaped payload must produce None cumulative_total_tokens" + ); + } + + #[test] + fn cumulative_total_absent_on_baseline_leaves_turn_total_null_on_second_turn() { + // Baseline was set without a total (e.g. first goose turn); second + // turn reports a total. No baseline to diff against → turn total None. + let mut tracker = UsageTracker::default(); + tracker.begin_turn("sess-t5"); + tracker.record("sess-t5", &payload_with_total(100, 20, None)); // no total + let _ = tracker.take(); + + tracker.begin_turn("sess-t5"); + tracker.record("sess-t5", &payload_with_total(200, 50, Some(250))); + let usage = tracker.take().expect("pending"); + + assert!(usage.delta_reliable, "input/output delta unaffected"); + assert!( + usage.turn_total_tokens.is_none(), + "absent baseline total → turn total null even when current has a total" + ); + assert_eq!(usage.cumulative_total_tokens, Some(250)); + } } diff --git a/crates/buzz-agent/src/agent.rs b/crates/buzz-agent/src/agent.rs index ed04daca2c..ce4d2db4b3 100644 --- a/crates/buzz-agent/src/agent.rs +++ b/crates/buzz-agent/src/agent.rs @@ -14,7 +14,7 @@ use crate::mcp::ResultBudget; use crate::types::{ AgentError, ContentBlock, HistoryItem, ProviderStop, StopReason, ToolCall, ToolResult, - ToolResultContent, + ToolResultContent, TurnTotalState, }; use crate::wire::{self, WireSender}; @@ -65,6 +65,17 @@ pub struct RunCtx<'a> { /// Consumers price this slice at the provider's cached rate; without it /// every round of a growing conversation is billed at full price. pub turn_cached_input_tokens: &'a mut Option, + /// Tri-state total-token accumulator for this turn. + /// + /// - `Unseen`: no usage-bearing response observed yet this turn (initial state). + /// - `Exact(n)`: every usage-bearing response so far reported a genuine + /// provider total; `n` is their sum. + /// - `Unknown`: at least one usage-bearing response lacked a provider total; + /// this turn can never produce a reliable total. + /// + /// Reset to `Unseen` at turn start in `run()`. Callers must not derive a + /// total by summing input+output — that is the UI display approximation only. + pub turn_total_state: &'a mut TurnTotalState, } impl RunCtx<'_> { @@ -84,6 +95,7 @@ impl RunCtx<'_> { *self.turn_input_tokens = None; *self.turn_output_tokens = None; *self.turn_cached_input_tokens = None; + *self.turn_total_state = TurnTotalState::Unseen; let mut round = 0u32; // Per-prompt `_Stop` objection count. Bounded per prompt (not per @@ -192,6 +204,20 @@ impl RunCtx<'_> { .saturating_add(cached), ); } + // Fold the provider-reported total into the turn tri-state, but only + // when this response was usage-bearing (had input or output tokens). + // A response with no usage at all is not evidence of a missing total + // and must not poison the accumulator. + // + // Shape assumption: documented OpenAI-compatible responses that carry + // `total_tokens` always co-report at least one of `prompt_tokens` / + // `completion_tokens`. A response that supplies only `total_tokens` + // with neither category is therefore not a supported shape and would + // be silently ignored here. If that shape is ever encountered, extend + // this gate rather than representing absent categories as zero. + if response.input_tokens.is_some() || response.output_tokens.is_some() { + *self.turn_total_state = self.turn_total_state.fold(response.total_tokens); + } if !response.reasoning.is_empty() { wire::send( diff --git a/crates/buzz-agent/src/lib.rs b/crates/buzz-agent/src/lib.rs index 6745dd0f92..9a45bf4c98 100644 --- a/crates/buzz-agent/src/lib.rs +++ b/crates/buzz-agent/src/lib.rs @@ -105,6 +105,14 @@ struct Session { /// it so a consumer can price the cached slice at the provider's discounted /// rate instead of assuming every input token cost full price. accumulated_cached_input_tokens: u64, + /// Session-cumulative total-token state across all turns. + /// + /// Mirrors the per-turn `TurnTotalState` tri-state: starts `Unseen`, + /// becomes `Exact(n)` as turns with genuine provider totals complete, + /// transitions permanently to `Unknown` when any turn lacks a total or + /// when the cumulative would otherwise decrease. Only emitted in the + /// `usage_update` notification when `Exact`. + accumulated_total_state: crate::types::TurnTotalState, } fn die(msg: String) -> ! { @@ -432,6 +440,7 @@ async fn session_new(app: &Arc, id: Value, params: Value, wire_tx: &WireSen accumulated_input_tokens: 0, accumulated_output_tokens: 0, accumulated_cached_input_tokens: 0, + accumulated_total_state: crate::types::TurnTotalState::Unseen, }, ); drop(sessions); @@ -679,6 +688,7 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender let mut turn_input_tokens: Option = None; let mut turn_output_tokens: Option = None; let mut turn_cached_input_tokens: Option = None; + let mut turn_total_state = crate::types::TurnTotalState::Unseen; let mut ctx = RunCtx { cfg: &app.cfg, effective_model: effective_model_str, @@ -698,6 +708,7 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender turn_input_tokens: &mut turn_input_tokens, turn_output_tokens: &mut turn_output_tokens, turn_cached_input_tokens: &mut turn_cached_input_tokens, + turn_total_state: &mut turn_total_state, }; let result = ctx.run(p.prompt).await; if let Some(s) = app.sessions.lock().await.get_mut(&sid) { @@ -733,10 +744,18 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender s.accumulated_cached_input_tokens = s .accumulated_cached_input_tokens .saturating_add(turn_cached_input_tokens.unwrap_or(0)); + // Fold the per-turn total state into the session cumulative. + // Unknown poisons the session permanently; Exact adds to running sum; + // Unseen (turn emitted no usage) leaves the cumulative unchanged. + // Uses TurnTotalState::merge_session, which applies the same + // checked-add / overflow-poisons contract as the per-response fold. + s.accumulated_total_state = + s.accumulated_total_state.merge_session(turn_total_state); Some(( s.accumulated_input_tokens, s.accumulated_output_tokens, s.accumulated_cached_input_tokens, + s.accumulated_total_state, )) } else { // Session is gone — the accumulated baseline no longer exists, so @@ -744,29 +763,32 @@ async fn run_prompt(app: Arc, id: Value, params: Value, wire_tx: WireSender None } }; - if let Some((accumulated_in, accumulated_out, accumulated_cached)) = accumulated { - wire::send( - &wire_tx, - goose_session_update( - &sid, - json!({ - "sessionUpdate": "usage_update", - // used: total tokens as a context-usage proxy; - // contextLimit: 0 (buzz-agent has no context limit tracking). - "used": accumulated_in.saturating_add(accumulated_out), - "contextLimit": 0u64, - "accumulatedInputTokens": accumulated_in, - "accumulatedOutputTokens": accumulated_out, - // A subset of accumulatedInputTokens, not an addition to - // it. Extends goose's usage_update shape; a consumer that - // does not know the field ignores it and prices exactly as - // it did before. - "accumulatedCachedInputTokens": accumulated_cached, - "model": effective_model_str, - }), - ), - ) - .await; + if let Some((accumulated_in, accumulated_out, accumulated_cached, accumulated_total)) = + accumulated + { + // Build the usage_update payload. `accumulatedTotalTokens` is only + // included when the cumulative is exactly known — never when Unseen + // (no total ever observed) or Unknown (at least one turn lacked a + // total). A goose consumer that doesn't recognise the field ignores it. + let mut update = serde_json::json!({ + "sessionUpdate": "usage_update", + // used: total tokens as a context-usage proxy; + // contextLimit: 0 (buzz-agent has no context limit tracking). + "used": accumulated_in.saturating_add(accumulated_out), + "contextLimit": 0u64, + "accumulatedInputTokens": accumulated_in, + "accumulatedOutputTokens": accumulated_out, + // A subset of accumulatedInputTokens, not an addition to + // it. Extends goose's usage_update shape; a consumer that + // does not know the field ignores it and prices exactly as + // it did before. + "accumulatedCachedInputTokens": accumulated_cached, + "model": effective_model_str, + }); + if let crate::types::TurnTotalState::Exact(total) = accumulated_total { + update["accumulatedTotalTokens"] = serde_json::json!(total); + } + wire::send(&wire_tx, goose_session_update(&sid, update)).await; } } match result { diff --git a/crates/buzz-agent/src/llm.rs b/crates/buzz-agent/src/llm.rs index 22d3f8b73e..98b881eb45 100644 --- a/crates/buzz-agent/src/llm.rs +++ b/crates/buzz-agent/src/llm.rs @@ -1195,6 +1195,9 @@ fn parse_responses(v: Value) -> Result { &["cache_read_input_tokens"], &[("input_tokens_details", "cached_tokens")], ); + // Responses API reports a genuine provider total. Read it directly — + // never derived, so it stays None when the provider omits it. + let total_tokens = sum_usage(&v, &["total_tokens"]); Ok(LlmResponse { text, tool_calls, @@ -1202,6 +1205,7 @@ fn parse_responses(v: Value) -> Result { input_tokens, cached_input_tokens, output_tokens, + total_tokens, reasoning, }) } @@ -1434,6 +1438,9 @@ fn parse_anthropic(v: Value) -> Result { input_tokens, cached_input_tokens, output_tokens, + // Anthropic reports only category counts; NIP-AM forbids deriving a + // total from them. Always None for this provider. + total_tokens: None, reasoning, }) } @@ -1498,6 +1505,9 @@ fn parse_openai(v: Value) -> Result { let input_tokens = openai_chat_input_tokens(&v); let output_tokens = sum_usage(&v, &["completion_tokens"]); let cached_input_tokens = openai_chat_cached_tokens(&v); + // OpenAI Chat Completions reports a genuine provider total. Read it + // directly — never derived, so it stays None when the provider omits it. + let total_tokens = sum_usage(&v, &["total_tokens"]); Ok(LlmResponse { text, tool_calls, @@ -1505,6 +1515,7 @@ fn parse_openai(v: Value) -> Result { input_tokens, cached_input_tokens, output_tokens, + total_tokens, reasoning, }) } @@ -4072,6 +4083,85 @@ mod tests { assert_eq!(parse_openai(v).unwrap().input_tokens, None); } + // ── total_tokens parsing ─────────────────────────────────────────────── + + #[test] + fn parse_openai_chat_total_tokens_present_is_read() { + // Chat Completions: `usage.total_tokens` is a genuine provider total. + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "ok"}}], + "usage": {"prompt_tokens": 100, "completion_tokens": 25, "total_tokens": 125} + }); + let r = parse_openai(v).unwrap(); + assert_eq!(r.total_tokens, Some(125)); + assert_eq!(r.input_tokens, Some(100)); + assert_eq!(r.output_tokens, Some(25)); + } + + #[test] + fn parse_openai_chat_total_tokens_absent_is_none() { + // Chat Completions without `total_tokens` → None, not a derived sum. + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "ok"}}], + "usage": {"prompt_tokens": 100, "completion_tokens": 25} + }); + let r = parse_openai(v).unwrap(); + assert_eq!(r.total_tokens, None); + } + + #[test] + fn parse_responses_total_tokens_present_is_read() { + // Responses API: `usage.total_tokens` is a genuine provider total. + let v = serde_json::json!({ + "output": [{"type": "message", "content": [{"type": "output_text", "text": "hi"}]}], + "status": "completed", + "usage": {"input_tokens": 80, "output_tokens": 20, "total_tokens": 100} + }); + let r = parse_responses(v).unwrap(); + assert_eq!(r.total_tokens, Some(100)); + assert_eq!(r.input_tokens, Some(80)); + assert_eq!(r.output_tokens, Some(20)); + } + + #[test] + fn parse_responses_total_tokens_absent_is_none() { + // Responses API without `total_tokens` → None. + let v = serde_json::json!({ + "output": [{"type": "message", "content": [{"type": "output_text", "text": "hi"}]}], + "status": "completed", + "usage": {"input_tokens": 80, "output_tokens": 20} + }); + let r = parse_responses(v).unwrap(); + assert_eq!(r.total_tokens, None); + } + + #[test] + fn parse_anthropic_total_tokens_always_none() { + // Anthropic reports only category counts; NIP-AM forbids deriving a total. + // total_tokens must always be None regardless of what the response contains — + // including if a future Anthropic API version unexpectedly adds total_tokens. + let v = serde_json::json!({ + "content": [{"type": "text", "text": "hi"}], + "stop_reason": "end_turn", + "usage": { + "input_tokens": 100, + "cache_read_input_tokens": 50, + "cache_creation_input_tokens": 0, + "output_tokens": 30, + // Unexpected field: parse_anthropic must ignore this and return None. + "total_tokens": 180 + } + }); + let r = parse_anthropic(v).unwrap(); + assert!( + r.total_tokens.is_none(), + "Anthropic must never supply a total_tokens value" + ); + // Verify other fields still parse correctly. + assert_eq!(r.input_tokens, Some(150)); // inclusive sum with cache + assert_eq!(r.output_tokens, Some(30)); + } + #[test] fn parse_openai_reads_nested_cached_tokens() { // The shape vanilla OpenAI actually returns, captured from a live diff --git a/crates/buzz-agent/src/types.rs b/crates/buzz-agent/src/types.rs index a3d48a7cf1..31172def6d 100644 --- a/crates/buzz-agent/src/types.rs +++ b/crates/buzz-agent/src/types.rs @@ -172,6 +172,13 @@ pub struct LlmResponse { /// response carried no usage. Used to accumulate per-turn output counts /// for NIP-AM metric publishing. pub output_tokens: Option, + /// Provider-reported total tokens for this request, or `None` when the + /// provider does not report a genuine total. Present for OpenAI-shaped + /// responses (`usage.total_tokens`). Always `None` for Anthropic, which + /// reports only category counts; NIP-AM forbids summing categories into a + /// total. Callers must not derive this by summing `input_tokens + + /// output_tokens` — that is what the UI display approximation is for. + pub total_tokens: Option, /// Reasoning/thinking content emitted by the model before its answer, if /// any. Non-empty when the provider returns extended-thinking tokens: /// @@ -199,6 +206,94 @@ pub struct ToolDef { pub input_schema: Value, } +/// Tri-state accumulator for provider-reported total tokens within one ACP turn. +/// +/// Tracks whether every usage-bearing LLM response in the turn supplied a genuine +/// provider total. Used to accumulate a reliable per-turn total and contribute to +/// the session-cumulative total. +/// +/// - `Unseen`: no usage-bearing response observed yet (initial state for each turn). +/// - `Exact(n)`: every response so far reported a total; `n` is their sum. +/// - `Unknown`: at least one response lacked a total — permanently poisoned for +/// this turn. The session-cumulative also transitions to Unknown when any turn +/// lands Unknown, and stays there until a new session resets it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum TurnTotalState { + #[default] + Unseen, + Exact(u64), + Unknown, +} + +impl TurnTotalState { + /// Add two exact token counts with overflow protection. + /// + /// Returns `Exact(acc + n)` on success or `Unknown` on overflow. + /// This is the single implementation of the checked-add / overflow-poisons + /// contract; both `fold()` and `merge_session()` call this helper so a + /// change to overflow semantics needs to be made in exactly one place. + fn checked_exact_sum(acc: u64, n: u64) -> TurnTotalState { + match acc.checked_add(n) { + Some(sum) => TurnTotalState::Exact(sum), + None => TurnTotalState::Unknown, + } + } + + /// Fold one provider-reported total into the current state. + /// + /// `total`: `Some(n)` when the provider included a genuine total on this + /// response; `None` when it was absent (e.g. Anthropic, or an OpenAI + /// response that omits usage). Absence of a total on any usage-bearing + /// response poisons the whole turn. + /// + /// Overflow is handled by `checked_exact_sum`: a saturated value would + /// not be a genuine provider-reported total, so overflow → `Unknown`. + pub fn fold(self, total: Option) -> TurnTotalState { + match (self, total) { + // Already poisoned — stays Unknown regardless. + (TurnTotalState::Unknown, _) => TurnTotalState::Unknown, + // No total from this response — poison the accumulator. + (_, None) => TurnTotalState::Unknown, + // First response with a total. + (TurnTotalState::Unseen, Some(n)) => TurnTotalState::Exact(n), + // Subsequent response — delegate to the shared checked-sum helper. + (TurnTotalState::Exact(acc), Some(n)) => Self::checked_exact_sum(acc, n), + } + } + + /// Merge a completed turn's total state into the session-cumulative state. + /// + /// This is the turn→session boundary accumulation: + /// - An `Unseen` turn (no usage-bearing responses) leaves the cumulative unchanged. + /// - Any `Unknown` side poisons the session permanently. + /// - Two `Exact` values are summed via `checked_exact_sum`; overflow → `Unknown`. + /// + /// The checked-add logic lives in `checked_exact_sum`; both this function and + /// `fold()` call that helper so overflow semantics are defined once. + pub fn merge_session(self, turn: TurnTotalState) -> TurnTotalState { + match (self, turn) { + // Either side poisoned → session is poisoned. + (TurnTotalState::Unknown, _) | (_, TurnTotalState::Unknown) => TurnTotalState::Unknown, + // Turn had no usage-bearing responses → no change to cumulative. + (acc, TurnTotalState::Unseen) => acc, + // First exact turn — adopt its value. + (TurnTotalState::Unseen, TurnTotalState::Exact(n)) => TurnTotalState::Exact(n), + // Add to running exact sum — delegate to the shared checked-sum helper. + (TurnTotalState::Exact(acc), TurnTotalState::Exact(n)) => { + Self::checked_exact_sum(acc, n) + } + } + } + + /// Consume the exact value if present; `None` for `Unseen` or `Unknown`. + pub fn exact_value(self) -> Option { + match self { + TurnTotalState::Exact(n) => Some(n), + _ => None, + } + } +} + #[derive(Debug, Clone, Copy, PartialEq)] pub enum StopReason { EndTurn, @@ -413,3 +508,138 @@ mod tests { assert_eq!(item.estimated_bytes(), item.context_pressure_bytes()); } } + +#[cfg(test)] +mod turn_total_state_tests { + use super::TurnTotalState; + + // ── TurnTotalState::fold ─────────────────────────────────────────────── + + #[test] + fn fold_first_response_with_total_becomes_exact() { + let state = TurnTotalState::Unseen; + assert_eq!(state.fold(Some(100)), TurnTotalState::Exact(100)); + } + + #[test] + fn fold_first_response_without_total_becomes_unknown() { + // Missing total on any usage-bearing response poisons the turn. + let state = TurnTotalState::Unseen; + assert_eq!(state.fold(None), TurnTotalState::Unknown); + } + + #[test] + fn multiple_provider_rounds_all_with_totals_sum_correctly() { + // Multiple rounds all reporting a genuine total → Exact with their sum. + let state = TurnTotalState::Unseen; + let state = state.fold(Some(100)); + let state = state.fold(Some(50)); + let state = state.fold(Some(75)); + assert_eq!(state, TurnTotalState::Exact(225)); + } + + #[test] + fn mixed_present_and_missing_totals_within_one_turn_poisons_accumulator() { + // First round has a total, second does not → Unknown (permanently poisoned). + let state = TurnTotalState::Unseen; + let state = state.fold(Some(100)); // Exact(100) + let state = state.fold(None); // Missing → Unknown + assert_eq!(state, TurnTotalState::Unknown); + // Further rounds with totals don't un-poison. + let state = state.fold(Some(50)); + assert_eq!(state, TurnTotalState::Unknown); + } + + #[test] + fn unknown_stays_unknown_regardless_of_subsequent_totals() { + // Once poisoned, no subsequent total can recover the state. + let state = TurnTotalState::Unknown; + assert_eq!(state.fold(Some(999)), TurnTotalState::Unknown); + assert_eq!(state.fold(None), TurnTotalState::Unknown); + } + + #[test] + fn exact_value_returns_some_only_for_exact_variant() { + assert_eq!(TurnTotalState::Unseen.exact_value(), None); + assert_eq!(TurnTotalState::Unknown.exact_value(), None); + assert_eq!(TurnTotalState::Exact(42).exact_value(), Some(42)); + } + + #[test] + fn default_is_unseen() { + let state: TurnTotalState = Default::default(); + assert_eq!(state, TurnTotalState::Unseen); + } + + // ── overflow: fold ───────────────────────────────────────────────────── + + #[test] + fn fold_overflow_poisons_turn_not_saturates() { + // u64::MAX + 1 would saturate; checked_add must poison instead. + let state = TurnTotalState::Exact(u64::MAX); + assert_eq!( + state.fold(Some(1)), + TurnTotalState::Unknown, + "overflow in fold() must produce Unknown, not Exact(u64::MAX)" + ); + } + + // ── TurnTotalState::merge_session ────────────────────────────────────── + + #[test] + fn merge_session_unseen_turn_leaves_cumulative_unchanged() { + // An Unseen turn (no usage-bearing responses) must not alter the cumulative. + assert_eq!( + TurnTotalState::Exact(100).merge_session(TurnTotalState::Unseen), + TurnTotalState::Exact(100), + ); + assert_eq!( + TurnTotalState::Unseen.merge_session(TurnTotalState::Unseen), + TurnTotalState::Unseen, + ); + } + + #[test] + fn merge_session_exact_turn_adds_to_exact_cumulative() { + assert_eq!( + TurnTotalState::Exact(100).merge_session(TurnTotalState::Exact(50)), + TurnTotalState::Exact(150), + ); + } + + #[test] + fn merge_session_first_exact_turn_from_unseen_adopts_value() { + assert_eq!( + TurnTotalState::Unseen.merge_session(TurnTotalState::Exact(200)), + TurnTotalState::Exact(200), + ); + } + + #[test] + fn merge_session_unknown_turn_poisons_cumulative_permanently() { + assert_eq!( + TurnTotalState::Exact(100).merge_session(TurnTotalState::Unknown), + TurnTotalState::Unknown, + ); + // Poisoned session stays poisoned even with Unseen turn. + assert_eq!( + TurnTotalState::Unknown.merge_session(TurnTotalState::Unseen), + TurnTotalState::Unknown, + ); + // Poisoned session stays poisoned even with another Exact turn. + assert_eq!( + TurnTotalState::Unknown.merge_session(TurnTotalState::Exact(999)), + TurnTotalState::Unknown, + ); + } + + #[test] + fn merge_session_overflow_poisons_not_saturates() { + // Overflow at the session boundary must also produce Unknown. + assert_eq!( + TurnTotalState::Exact(u64::MAX).merge_session(TurnTotalState::Exact(1)), + TurnTotalState::Unknown, + "overflow in merge_session() must produce Unknown, not Exact(u64::MAX)" + ); + } +} diff --git a/crates/buzz-agent/tests/fake_llm.rs b/crates/buzz-agent/tests/fake_llm.rs index f2a86ac4b3..f782a9d476 100644 --- a/crates/buzz-agent/tests/fake_llm.rs +++ b/crates/buzz-agent/tests/fake_llm.rs @@ -771,6 +771,26 @@ fn openai_text_with_usage(content: &str, input_tokens: u64, output_tokens: u64) }) } +/// An OpenAI chat completion response WITH i/o usage but WITHOUT `total_tokens`. +/// Simulates a provider that omits the genuine total from its usage block. +/// buzz-agent must treat this turn's total as Unknown and poison the cumulative. +fn openai_text_with_usage_no_total(content: &str, input_tokens: u64, output_tokens: u64) -> Value { + json!({ + "id": "cc-nt", "object": "chat.completion", "model": "fake-model", + "choices": [{ + "index": 0, + "message": { "role": "assistant", "content": content }, + "finish_reason": "stop", + }], + "usage": { + "prompt_tokens": input_tokens, + "completion_tokens": output_tokens, + // total_tokens deliberately absent — simulates Anthropic or any + // provider that does not report a genuine total. + }, + }) +} + /// Returns true when `v` is a `_goose/unstable/session/update` usage_update /// notification. fn is_usage_update(v: &Value) -> bool { @@ -1123,3 +1143,164 @@ async fn steer_rejected_on_empty_prompt() { assert!(saw_reject, "empty steer prompt was not rejected"); h.shutdown().await; } + +// ─── Session-boundary total accumulation ──────────────────────────────────── + +/// Once a usage-bearing turn lacks a provider total, the session cumulative +/// becomes Unknown and `accumulatedTotalTokens` must be absent from subsequent +/// `usage_update` notifications — even if later turns supply a total. +/// +/// Sequence: turn 1 has total, turn 2 lacks total → session poisoned, turn 3 +/// has total → still poisoned. Only turn 1 must carry `accumulatedTotalTokens`. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn session_total_poisoned_by_missing_total_and_stays_poisoned() { + let url = spawn_fake_llm(vec![ + openai_text_with_usage("t1", 10, 5), // total present → Exact(15) + openai_text_with_usage_no_total("t2", 20, 8), // total absent → Unknown + openai_text_with_usage("t3", 15, 6), // total present → still Unknown + ]) + .await; + let mut h = Harness::spawn(&url).await; + let sid = init_session(&mut h).await; + + // ── Turn 1: total present ─────────────────────────────────────────────── + let p1 = h + .send( + "session/prompt", + json!({"sessionId": sid, "prompt": [{"type":"text","text":"t1"}]}), + ) + .await; + let (frames1, _) = recv_until_with_drain(&mut h, |v| v["id"] == p1).await; + let usage1 = frames1 + .iter() + .find(|v| is_usage_update(v)) + .expect("usage_update for turn 1"); + assert_eq!( + usage1["params"]["update"]["accumulatedTotalTokens"], + json!(15u64), + "turn 1 has genuine total; accumulatedTotalTokens must be 15" + ); + + // ── Turn 2: total absent — session is now poisoned ────────────────────── + let p2 = h + .send( + "session/prompt", + json!({"sessionId": sid, "prompt": [{"type":"text","text":"t2"}]}), + ) + .await; + let (frames2, _) = recv_until_with_drain(&mut h, |v| v["id"] == p2).await; + let usage2 = frames2 + .iter() + .find(|v| is_usage_update(v)) + .expect("usage_update for turn 2"); + assert!( + usage2["params"]["update"]["accumulatedTotalTokens"].is_null() + || usage2["params"]["update"] + .get("accumulatedTotalTokens") + .is_none(), + "turn 2 lacked total; accumulatedTotalTokens must be absent/null; got: {usage2:#?}" + ); + + // ── Turn 3: total present, but session is still poisoned ───────────────── + let p3 = h + .send( + "session/prompt", + json!({"sessionId": sid, "prompt": [{"type":"text","text":"t3"}]}), + ) + .await; + let (frames3, _) = recv_until_with_drain(&mut h, |v| v["id"] == p3).await; + let usage3 = frames3 + .iter() + .find(|v| is_usage_update(v)) + .expect("usage_update for turn 3"); + assert!( + usage3["params"]["update"]["accumulatedTotalTokens"].is_null() + || usage3["params"]["update"].get("accumulatedTotalTokens").is_none(), + "session is poisoned; accumulatedTotalTokens must remain absent even after a total-bearing turn; got: {usage3:#?}" + ); + + // i/o counters are unaffected by total poisoning. + assert_eq!( + usage3["params"]["update"]["accumulatedInputTokens"], + json!(45u64), + "poisoned total must not discard input accumulation" + ); + assert_eq!( + usage3["params"]["update"]["accumulatedOutputTokens"], + json!(19u64), + "poisoned total must not discard output accumulation" + ); + + h.shutdown().await; +} + +/// A new session starts fresh and can accumulate an exact total independently +/// of any previous session. This verifies `accumulated_total_state` is reset +/// to `Unseen` on `session/new`, not inherited from a prior session. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn new_session_resets_total_accumulation() { + // Session A: two turns both with totals → Exact should accumulate. + // Session B (new session/new call): starts fresh. + let url = spawn_fake_llm(vec![ + // Session A, turn 1 + openai_text_with_usage("s1t1", 10, 5), + // Session A, turn 2 + openai_text_with_usage("s1t2", 20, 8), + // Session B, turn 1 + openai_text_with_usage("s2t1", 30, 10), + ]) + .await; + let mut h = Harness::spawn(&url).await; + let sid_a = init_session(&mut h).await; + + // Session A, turn 1 + let p1 = h + .send( + "session/prompt", + json!({"sessionId": sid_a, "prompt": [{"type":"text","text":"s1t1"}]}), + ) + .await; + let (frames1, _) = recv_until_with_drain(&mut h, |v| v["id"] == p1).await; + let u1 = frames1.iter().find(|v| is_usage_update(v)).expect("usage1"); + assert_eq!( + u1["params"]["update"]["accumulatedTotalTokens"], + json!(15u64), + "session A turn 1 accumulated total" + ); + + // Session A, turn 2 — cumulative total is 15+28=43 + let p2 = h + .send( + "session/prompt", + json!({"sessionId": sid_a, "prompt": [{"type":"text","text":"s1t2"}]}), + ) + .await; + let (frames2, _) = recv_until_with_drain(&mut h, |v| v["id"] == p2).await; + let u2 = frames2.iter().find(|v| is_usage_update(v)).expect("usage2"); + assert_eq!( + u2["params"]["update"]["accumulatedTotalTokens"], + json!(43u64), + "session A turn 2 cumulative total must be 15+28=43" + ); + + // Start a new session — must reset accumulated_total_state to Unseen. + let sid_b = init_session(&mut h).await; + assert_ne!(sid_a, sid_b, "sessions must have distinct IDs"); + + // Session B, turn 1 — total 30+10=40. Must NOT start from 43. + let p3 = h + .send( + "session/prompt", + json!({"sessionId": sid_b, "prompt": [{"type":"text","text":"s2t1"}]}), + ) + .await; + let (frames3, _) = recv_until_with_drain(&mut h, |v| v["id"] == p3).await; + let u3 = frames3.iter().find(|v| is_usage_update(v)).expect("usage3"); + assert_eq!( + u3["params"]["update"]["accumulatedTotalTokens"], + json!(40u64), + "new session must start fresh — accumulated total must be 40, not 83" + ); + + h.shutdown().await; +} From ab55fee81896d2b03edf5d2ca5012b715be2b93d Mon Sep 17 00:00:00 2001 From: Will Pfleger Date: Wed, 29 Jul 2026 19:08:35 -0400 Subject: [PATCH 27/32] feat: add first-class OpenRouter provider support (#1975) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary First-class `Provider::OpenRouter` support joining the existing anthropic/openai/databricks providers. Reuses the Chat Completions path with targeted mutations for OpenRouter's routing contract. **Core (`crates/buzz-agent`):** - `Provider::OpenRouter` enum variant with `OPENROUTER_API_KEY`, `BUZZ_AGENT_MODEL` → `OPENROUTER_MODEL` fallback, `OPENROUTER_BASE_URL` env convention - Body mutator: `reasoning: {effort}` when effort is configured, and `max_completion_tokens` translated to OpenRouter's `max_tokens` spelling; no `provider.require_parameters` filter (it routes only to endpoints advertising every parameter in the body, which hard-404s a valid model id); summaries get neither. `openai_body` is always called with `effort=None` on the OpenRouter path — the `reasoning` object is added by the mutator directly, so `reasoning_effort` is structurally absent. - Attribution headers: `HTTP-Referer: https://github.com/block/buzz`, `X-OpenRouter-Title: Buzz` - Error-inside-200 check in shared `parse_openai` (`finish_reason == "error"`) - 401 auth handling: static API keys (`refresh_now` returns the same token) fail terminal immediately with one wire request; PKCE/minting sources get one retry with the fresh token. - Status+`error_type` retry matrix (4-arm collapsed form): 429 (honor `Retry-After`), 502 (retry), 503/`provider_overloaded` (honor `Retry-After`), everything else including untyped 503 (bounded retries → actionable routing message). 499 included matching shared `post()` (#2175) for turn-timeout stall surfacing. Terminal failures wrapped in `terminal_llm_error` for duration+attempt-count context. - `anthropic/*` `cache_control` injection (model-gated, mixed-content safe) - Provider-agnostic `reasoning_details` opaque round-trip on `HistoryItem::Assistant` for tool-call continuations — captured verbatim in `parse_openai_with_reasoning_details`, replayed verbatim in `openai_body`, byte-accounting charged. `provider_extra` passthrough from `make_tool_call` composes independently. **Desktop:** - Readiness arms checking `OPENROUTER_API_KEY` + `OPENROUTER_MODEL` - Model discovery via `{OPENROUTER_BASE_URL}/models` filtered on `supported_parameters` contains `tools` - Picker entry, credential config, effort table 3-file sync **`desktop/src/features/agents/AGENTS.md`: no rules changed** — the scoped rule requiring an explicit note is satisfied here. Implements the gate-cleared plan from `PLANS/OPENROUTER_PROVIDER_PLAN.md` (rev 3). Signed-off-by: Will Pfleger Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 --- crates/buzz-agent/README.md | 30 +- crates/buzz-agent/src/agent.rs | 76 + crates/buzz-agent/src/config.rs | 60 +- crates/buzz-agent/src/handoff.rs | 6 +- crates/buzz-agent/src/llm.rs | 1963 ++++++++++++++++- crates/buzz-agent/src/types.rs | 18 +- .../src-tauri/src/commands/agent_models.rs | 126 +- .../commands/agent_models_discovery_config.rs | 112 + .../src/commands/agent_models_openrouter.rs | 112 + .../src/commands/agent_models_tests.rs | 291 +++ .../src-tauri/src/managed_agents/readiness.rs | 229 +- .../readiness_goose_file_config_tests.rs | 190 ++ .../features/agents/ui/agentConfigOptions.tsx | 9 +- .../src/features/agents/ui/buzzAgentConfig.ts | 3 + .../agents/ui/effortTable.fixture.json | 7 + 15 files changed, 2943 insertions(+), 289 deletions(-) create mode 100644 desktop/src-tauri/src/commands/agent_models_discovery_config.rs create mode 100644 desktop/src-tauri/src/commands/agent_models_openrouter.rs create mode 100644 desktop/src-tauri/src/managed_agents/readiness_goose_file_config_tests.rs diff --git a/crates/buzz-agent/README.md b/crates/buzz-agent/README.md index a2504db451..f138e4a4f1 100644 --- a/crates/buzz-agent/README.md +++ b/crates/buzz-agent/README.md @@ -21,9 +21,9 @@ HTTPS │ ▼ - Anthropic Messages API - or any OpenAI-compat - (vLLM, llama.cpp, OpenRouter, + Anthropic Messages API, + OpenRouter, or any OpenAI-compat + (vLLM, llama.cpp, Databricks, Block Gateway, Ollama, …) ``` @@ -50,6 +50,12 @@ OPENAI_COMPAT_MODEL=gpt-5 \ OPENAI_COMPAT_BASE_URL=https://api.openai.com/v1 \ ./target/release/buzz-agent +# Or OpenRouter +BUZZ_AGENT_PROVIDER=openrouter \ +OPENROUTER_API_KEY=sk-or-v1-... \ +OPENROUTER_MODEL=anthropic/claude-sonnet-4.5 \ + ./target/release/buzz-agent + # Or Databricks model serving via OAuth 2.0 PKCE BUZZ_AGENT_PROVIDER=databricks \ DATABRICKS_HOST=https://dbc-...cloud.databricks.com \ @@ -129,15 +135,18 @@ Everything is environment variables. No flags, no config files. (We are a subpro | Variable | Default | Notes | |---|---|---| -| `BUZZ_AGENT_PROVIDER` | — | Required. `anthropic`, `openai`, `databricks`, or `databricks_v2`. No implicit fallback — the agent errors at startup when this is unset. | +| `BUZZ_AGENT_PROVIDER` | — | Required. `anthropic`, `openai`, `openrouter`, `databricks`, or `databricks_v2`. No implicit fallback — the agent errors at startup when this is unset. | | `ANTHROPIC_API_KEY` | — | Required when provider=anthropic. | | `ANTHROPIC_MODEL` | — | Required when provider=anthropic. | | `ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | | | `ANTHROPIC_API_VERSION` | `2023-06-01` | | | `OPENAI_COMPAT_API_KEY` | — | Required when provider=openai. | | `OPENAI_COMPAT_MODEL` | — | Required when provider=openai. | -| `OPENAI_COMPAT_BASE_URL` | `https://api.openai.com/v1` | Point at vLLM, llama.cpp, OpenRouter, Ollama, etc. | +| `OPENAI_COMPAT_BASE_URL` | `https://api.openai.com/v1` | Point at vLLM, llama.cpp, Ollama, etc. | | `OPENAI_COMPAT_API` | `auto` | `auto` \| `chat` \| `responses`. `auto` picks Responses for `*.openai.com`, Chat Completions everywhere else. | +| `OPENROUTER_API_KEY` | — | Required when provider=openrouter. | +| `OPENROUTER_MODEL` | — | Required when provider=openrouter. Use OpenRouter's `vendor/model` id, e.g. `anthropic/claude-sonnet-4.5`. | +| `OPENROUTER_BASE_URL` | `https://openrouter.ai/api/v1` | | | `DATABRICKS_HOST` | — | Required when provider=databricks or provider=databricks_v2. | | `DATABRICKS_MODEL` | — | Required when provider=databricks or provider=databricks_v2. | | `DATABRICKS_TOKEN` | — | Optional static bearer escape hatch. If unset, Databricks uses browser OAuth + refresh cache. | @@ -167,17 +176,24 @@ Everything is environment variables. No flags, no config files. (We are a subpro | vLLM | `openai` | `POST {base}/chat/completions` | any tool-calling model | | llama.cpp | `openai` | `POST {base}/chat/completions` | any tool-calling GGUF | | Ollama | `openai` | `POST {base}/chat/completions` | llama3.1, qwen2.5-coder | -| OpenRouter | `openai` | `POST {base}/chat/completions` | anything they route | | Block Gateway | `openai` | `POST {base}/chat/completions` | gpt-5, claude | +| OpenRouter | `openrouter` | `POST {base}/chat/completions` | anything they route (extended-thinking replay, provider-agnostic tool calling) | | Databricks | `databricks` | `POST {host}/serving-endpoints/{model}/invocations` | goose-claude-4-6-sonnet | | Databricks AI Gateway v2 | `databricks_v2` | `POST {host}/ai-gateway/{provider}/v1/...` | databricks-gpt-5-5, databricks-claude-opus-4-7 | -If `BUZZ_AGENT_PROVIDER=anthropic` is selected without `ANTHROPIC_API_KEY`, or `BUZZ_AGENT_PROVIDER=openai` is selected without `OPENAI_COMPAT_API_KEY`, the agent returns an error — there is no implicit fallback to another provider. +If `BUZZ_AGENT_PROVIDER=anthropic` is selected without `ANTHROPIC_API_KEY`, `BUZZ_AGENT_PROVIDER=openai` is selected without `OPENAI_COMPAT_API_KEY`, or `BUZZ_AGENT_PROVIDER=openrouter` is selected without `OPENROUTER_API_KEY`, the agent returns an error — there is no implicit fallback to another provider. `provider=openai` speaks two HTTP dialects: the [Responses API](https://platform.openai.com/docs/api-reference/responses) (`/v1/responses`, required for GPT-5 / o-series tool-calling on OpenAI's own service) and the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat) (`/chat/completions`, the broadly-supported OpenAI-compatible wire format). By default (`OPENAI_COMPAT_API=auto`) the agent picks **Responses** when `OPENAI_COMPAT_BASE_URL` points at an `*.openai.com` host and **Chat Completions** everywhere else. Pin the choice explicitly with `OPENAI_COMPAT_API=chat` or `OPENAI_COMPAT_API=responses` for providers that diverge from the default (e.g. a Responses-compatible self-hosted gateway). +`provider=openrouter` is first-class, not routed through `provider=openai`: it speaks OpenAI's Chat Completions wire format but with OpenRouter-specific extensions layered on top — + +- `reasoning.effort` is set on the request when reasoning effort is configured. The request deliberately carries no `provider.require_parameters` filter: that filter routes only to endpoints advertising every parameter in the body, and 83 of 274 tools-capable OpenRouter models do not advertise `reasoning`, so it turns an effort setting into a hard 404 on a valid model id. A model that cannot reason answers without reasoning instead. +- The response's `reasoning_details` array (opaque extended-thinking payload) is captured and replayed byte-for-byte on the next turn's assistant message, so multi-turn tool use keeps the model's chain-of-thought. +- `anthropic/*` models get Anthropic-style `cache_control` breakpoints injected on the system message and the last two user messages. +- Retryable statuses (429 and typed `provider_overloaded` 503) honor the documented `Retry-After` header (clamped to a small ceiling — see `RETRY_AFTER_CAP_SECS` in `llm.rs` — since the sleep happens outside `BUZZ_AGENT_LLM_TIMEOUT_SECS`); 502 and untyped 503 retry with jittered backoff instead. `401` is treated as an expired/invalid key and refreshed once, while `402` (no credits) and `403` (guardrail/moderation/permission) fail immediately without retry. + `Provider` is a Rust `enum` with one `match` in `Llm::complete`. There is no trait, no `Box`, no async-trait. Adding a provider is a `match` arm and one `body`/`parse` pair in `llm.rs`. ## MCP Servers diff --git a/crates/buzz-agent/src/agent.rs b/crates/buzz-agent/src/agent.rs index ce4d2db4b3..48d4ea3b02 100644 --- a/crates/buzz-agent/src/agent.rs +++ b/crates/buzz-agent/src/agent.rs @@ -256,6 +256,7 @@ impl RunCtx<'_> { self.history.push(HistoryItem::Assistant { text: response.text, tool_calls: Vec::new(), + reasoning_details: response.reasoning_details.clone(), }); let stop = map_stop(response.stop); // Only gate genuine end_turn — don't override max_tokens/refusal. @@ -292,6 +293,7 @@ impl RunCtx<'_> { self.history.push(HistoryItem::Assistant { text: response.text, tool_calls: calls.clone(), + reasoning_details: response.reasoning_details, }); if let Some(stop) = self.execute_calls(&calls).await { @@ -726,6 +728,7 @@ pub(crate) fn push_hook_outputs_as_tool_results( // preserve. provider_extra: Default::default(), }], + reasoning_details: None, }); history.push(HistoryItem::ToolResult(ToolResult { provider_id, @@ -790,3 +793,76 @@ fn map_stop(p: ProviderStop) -> StopReason { ProviderStop::Refusal => StopReason::Refusal, } } + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + /// A9 regression: `reasoning_details` contributes real bytes to + /// `estimated_bytes` (see `types.rs::HistoryItem::size_with`), so a + /// history item carrying a large opaque reasoning array must actually + /// drive `truncate_history` eviction — not be silently invisible to the + /// sizing gate that decides what survives a turn. + #[test] + fn truncate_history_evicts_oldest_turn_with_reasoning_details() { + let big_reasoning = json!([{ "type": "reasoning.text", "text": "x".repeat(400) }]); + let mut history = vec![ + HistoryItem::User("first question".into()), + HistoryItem::Assistant { + text: "first answer".into(), + tool_calls: vec![], + reasoning_details: Some(big_reasoning), + }, + HistoryItem::User("second question".into()), + HistoryItem::Assistant { + text: "second answer".into(), + tool_calls: vec![], + reasoning_details: None, + }, + ]; + + let total_before: usize = history.iter().map(HistoryItem::estimated_bytes).sum(); + // Budget below the total but above the second (smaller) turn alone, + // so only the oldest user+assistant pair — the one carrying + // reasoning_details — must be dropped. + let max_bytes = total_before - 100; + assert!( + max_bytes > 0, + "test fixture must leave room to evict only one turn" + ); + + truncate_history(&mut history, max_bytes); + + assert_eq!( + history.len(), + 2, + "the oldest user+assistant turn (with reasoning_details) must be evicted" + ); + assert!(matches!(&history[0], HistoryItem::User(s) if s == "second question")); + assert!( + matches!(&history[1], HistoryItem::Assistant { text, .. } if text == "second answer") + ); + let total_after: usize = history.iter().map(HistoryItem::estimated_bytes).sum(); + assert!(total_after <= max_bytes); + } + + #[test] + fn truncate_history_noop_when_under_budget() { + let mut history = vec![ + HistoryItem::User("hi".into()), + HistoryItem::Assistant { + text: "hello".into(), + tool_calls: vec![], + reasoning_details: None, + }, + ]; + let original_len = history.len(); + truncate_history(&mut history, 1_000_000); + assert_eq!( + history.len(), + original_len, + "under budget must not evict anything" + ); + } +} diff --git a/crates/buzz-agent/src/config.rs b/crates/buzz-agent/src/config.rs index 037b67b3cb..a0e64f1a9d 100644 --- a/crates/buzz-agent/src/config.rs +++ b/crates/buzz-agent/src/config.rs @@ -671,6 +671,8 @@ pub enum Provider { /// Databricks AI Gateway v2. Routes by model family through the gateway's /// OpenAI Responses, Anthropic Messages, or MLflow Chat Completions paths. DatabricksV2, + /// OpenRouter multi-provider gateway. Routes to `{base_url}/chat/completions` with bearer auth. Wire format is OpenAI-chat-compatible. + OpenRouter, } /// Which OpenAI-family HTTP API to call. Set via `OPENAI_COMPAT_API` @@ -740,10 +742,11 @@ pub struct Config { pub thinking_effort: Option, /// Emit Anthropic `cache_control` breakpoints on the stable prefix /// (tools + system prompt) and the rolling conversation tail. Default on; - /// disable with `BUZZ_AGENT_PROMPT_CACHING=0`. Only consulted on Anthropic - /// Messages routes (first-party Anthropic and the DatabricksV2 Claude - /// route) — the Databricks gateway does not auto-cache, so without this the - /// surfaced `cache_read_input_tokens` is structurally always 0. + /// disable with `BUZZ_AGENT_PROMPT_CACHING=0`. Consulted on every route that + /// speaks the Anthropic caching dialect: first-party Anthropic, the + /// DatabricksV2 Claude route, and OpenRouter's `anthropic/*` models. The + /// Databricks gateway does not auto-cache, so without this the surfaced + /// `cache_read_input_tokens` is structurally always 0. pub prompt_caching: bool, } @@ -755,6 +758,7 @@ impl Config { env("BUZZ_AGENT_PROVIDER").as_deref(), env("ANTHROPIC_API_KEY").as_deref(), env("OPENAI_COMPAT_API_KEY").as_deref(), + env("OPENROUTER_API_KEY").as_deref(), )?; // Universal model override — takes priority over provider-specific model @@ -797,6 +801,16 @@ impl Config { databricks_host.ok_or_else(|| "config: DATABRICKS_HOST required".to_string())?, OpenAiApi::Chat, // only read by OpenAI/legacy Databricks dispatch ), + Provider::OpenRouter => ( + req("OPENROUTER_API_KEY")?, + resolve_model( + buzz_agent_model.as_deref(), + env("OPENROUTER_MODEL").as_deref(), + ) + .ok_or_else(|| "config: OPENROUTER_MODEL required".to_string())?, + env_or("OPENROUTER_BASE_URL", "https://openrouter.ai/api/v1"), + OpenAiApi::Chat, // OpenRouter uses Chat Completions only + ), }; let system_prompt = match (env("BUZZ_AGENT_SYSTEM_PROMPT"), env("BUZZ_AGENT_SYSTEM_PROMPT_FILE")) { (Some(_), Some(_)) => return Err( @@ -1002,6 +1016,7 @@ fn resolve_provider( requested: Option<&str>, anthropic_key: Option<&str>, openai_key: Option<&str>, + openrouter_key: Option<&str>, ) -> Result { match requested.map(str::trim).filter(|s| !s.is_empty()) { Some(raw) => { @@ -1017,6 +1032,8 @@ fn resolve_provider( ), "databricks" => Ok(Provider::Databricks), "databricks_v2" | "databricks-v2" => Ok(Provider::DatabricksV2), + "openrouter" if present_nonempty(openrouter_key) => Ok(Provider::OpenRouter), + "openrouter" => Err("config: OPENROUTER_API_KEY required".into()), _ => Err(format!( "config: BUZZ_AGENT_PROVIDER={raw} not supported" )), @@ -1234,11 +1251,11 @@ mod tests { #[test] fn resolve_provider_keeps_requested_provider_when_token_present() { assert_eq!( - resolve_provider(Some("anthropic"), Some("sk-ant"), None,).unwrap(), + resolve_provider(Some("anthropic"), Some("sk-ant"), None, None).unwrap(), Provider::Anthropic ); assert_eq!( - resolve_provider(Some("openai"), None, Some("sk-openai"),).unwrap(), + resolve_provider(Some("openai"), None, Some("sk-openai"), None).unwrap(), Provider::OpenAi ); } @@ -1246,17 +1263,17 @@ mod tests { #[test] fn resolve_provider_errors_when_requested_provider_key_missing() { // No fallback — missing key returns an error regardless of Databricks availability. - let err = resolve_provider(Some("anthropic"), None, None).unwrap_err(); + let err = resolve_provider(Some("anthropic"), None, None, None).unwrap_err(); assert!(err.contains("ANTHROPIC_API_KEY required"), "{err}"); - let err = resolve_provider(Some("openai-compat"), None, Some(" ")).unwrap_err(); + let err = resolve_provider(Some("openai-compat"), None, Some(" "), None).unwrap_err(); assert!(err.contains("OPENAI_COMPAT_API_KEY required"), "{err}"); } #[test] fn resolve_provider_errors_when_provider_env_absent() { // No implicit inference — absent BUZZ_AGENT_PROVIDER is an error. - let err = resolve_provider(None, None, None).unwrap_err(); + let err = resolve_provider(None, None, None, None).unwrap_err(); assert!(err.contains("BUZZ_AGENT_PROVIDER is required"), "{err}"); } @@ -1266,19 +1283,19 @@ mod tests { // When BUZZ_AGENT_PROVIDER=databricks, resolve_provider succeeds regardless // of DATABRICKS_HOST/MODEL (those are validated later in from_env()). assert_eq!( - resolve_provider(Some("databricks"), None, None).unwrap(), + resolve_provider(Some("databricks"), None, None, None).unwrap(), Provider::Databricks ); // Missing key for other providers still errors — no Databricks fallback. - let err = resolve_provider(Some("openai"), None, None).unwrap_err(); + let err = resolve_provider(Some("openai"), None, None, None).unwrap_err(); assert!(err.contains("OPENAI_COMPAT_API_KEY required"), "{err}"); - let err = resolve_provider(None, None, None).unwrap_err(); + let err = resolve_provider(None, None, None, None).unwrap_err(); assert!(err.contains("BUZZ_AGENT_PROVIDER is required"), "{err}"); } #[test] fn resolve_provider_unsupported_error_preserves_user_casing() { - let err = resolve_provider(Some("OpenAIish"), None, None).unwrap_err(); + let err = resolve_provider(Some("OpenAIish"), None, None, None).unwrap_err(); assert!(err.contains("BUZZ_AGENT_PROVIDER=OpenAIish")); } @@ -2666,6 +2683,9 @@ mod tests { if p == "databricks" { return openai_result(&m); } + if p == "openrouter" { + return (ALL_7.to_vec(), Some("medium")); + } // openai-compat, unknown, empty → all-7, default medium. (ALL_7.to_vec(), Some("medium")) } @@ -2717,4 +2737,18 @@ mod tests { ); } } + + #[test] + fn resolve_provider_openrouter_with_key() { + assert_eq!( + resolve_provider(Some("openrouter"), None, None, Some("sk-or-123")).unwrap(), + Provider::OpenRouter + ); + } + + #[test] + fn resolve_provider_openrouter_missing_key() { + let err = resolve_provider(Some("openrouter"), None, None, None).unwrap_err(); + assert!(err.contains("OPENROUTER_API_KEY")); + } } diff --git a/crates/buzz-agent/src/handoff.rs b/crates/buzz-agent/src/handoff.rs index 9c27c6606d..3b0feefecf 100644 --- a/crates/buzz-agent/src/handoff.rs +++ b/crates/buzz-agent/src/handoff.rs @@ -259,7 +259,11 @@ fn push_history_snippet(out: &mut String, item: &HistoryItem) { out.push_str(s); out.push('\n'); } - HistoryItem::Assistant { text, tool_calls } => { + HistoryItem::Assistant { + text, + tool_calls, + reasoning_details: _, + } => { out.push_str("[assistant] "); if !text.is_empty() { out.push_str(text); diff --git a/crates/buzz-agent/src/llm.rs b/crates/buzz-agent/src/llm.rs index 98b881eb45..f595a165e5 100644 --- a/crates/buzz-agent/src/llm.rs +++ b/crates/buzz-agent/src/llm.rs @@ -146,6 +146,18 @@ impl Llm { .await?; parse_anthropic(v) } + Provider::OpenRouter => { + let mut body = + openai_body(cfg, system_prompt, history, tools, effective_model, None); + apply_openrouter_mutations( + &mut body, + cfg.thinking_effort, + effective_model, + cfg.prompt_caching, + ); + let v = self.post_openrouter(cfg, &body).await?; + parse_openai_with_reasoning_details(v) + } Provider::OpenAi | Provider::Databricks => { self.openai_request( cfg, @@ -248,6 +260,16 @@ impl Llm { }); Ok(parse_anthropic(self.post_anthropic(cfg, &body).await?)?.text) } + Provider::OpenRouter => { + let body = openrouter_summary_body( + effective_model, + system_prompt, + user_prompt, + max_output_tokens, + ); + let v = self.post_openrouter(cfg, &body).await?; + Ok(parse_openai(v)?.text) + } Provider::OpenAi | Provider::Databricks => { let r = self .openai_request( @@ -652,6 +674,32 @@ impl Llm { } } + async fn post_openrouter(&self, cfg: &Config, body: &Value) -> Result { + let url = format!("{}/chat/completions", cfg.base_url.trim_end_matches('/')); + let mut bearer = self.auth.bearer().await?; + let mut refreshed = false; + loop { + match openrouter_post(&self.http, &url, body, &bearer).await { + Err(AgentError::LlmAuth(_)) if !refreshed => { + refreshed = true; + let new_bearer = self.auth.refresh_now(&bearer).await?; + // A static key refreshes to itself — a byte-identical retry + // would be a guaranteed duplicate request against a key the + // server just rejected. Fail terminal immediately; the retry + // is only meaningful when the source can actually mint a + // distinct token (e.g., a PKCE OAuth source). + if new_bearer == bearer { + return Err(AgentError::LlmAuth( + "401: static key rejected — update key in agent settings".into(), + )); + } + bearer = new_bearer; + } + result => return result, + } + } + } + /// If `err` names `/v1/responses` / "use the Responses API", latch a /// sticky upgrade so subsequent OpenAI calls hit Responses. Logged once. fn try_upgrade(&self, err: &AgentError) -> bool { @@ -695,7 +743,11 @@ fn anthropic_body( messages.push(json!({ "role": "user", "content": [{ "type": "text", "text": text }] })); } - HistoryItem::Assistant { text, tool_calls } => { + HistoryItem::Assistant { + text, + tool_calls, + reasoning_details: _, + } => { flush(&mut messages, &mut pending); let mut content: Vec = Vec::new(); if !text.is_empty() { @@ -839,11 +891,18 @@ fn openai_body( flush_images(&mut messages, &mut pending_images); messages.push(json!({ "role": "user", "content": text })); } - HistoryItem::Assistant { text, tool_calls } => { + HistoryItem::Assistant { + text, + tool_calls, + reasoning_details, + } => { flush_images(&mut messages, &mut pending_images); let mut msg = serde_json::Map::new(); msg.insert("role".into(), json!("assistant")); msg.insert("content".into(), json!(text.as_str())); + if let Some(details) = reasoning_details { + msg.insert("reasoning_details".into(), details.clone()); + } if !tool_calls.is_empty() { let calls: Vec = tool_calls .iter() @@ -951,7 +1010,11 @@ fn responses_body( "role": "user", "content": [{ "type": "input_text", "text": text }], })), - HistoryItem::Assistant { text, tool_calls } => { + HistoryItem::Assistant { + text, + tool_calls, + reasoning_details: _, + } => { if !text.is_empty() { input.push(json!({ "role": "assistant", @@ -1207,6 +1270,7 @@ fn parse_responses(v: Value) -> Result { output_tokens, total_tokens, reasoning, + reasoning_details: None, }) } @@ -1442,10 +1506,44 @@ fn parse_anthropic(v: Value) -> Result { // total from them. Always None for this provider. total_tokens: None, reasoning, + reasoning_details: None, }) } fn parse_openai(v: Value) -> Result { + // A5: error-inside-200 check — choice-level `finish_reason == "error"` + if let Some(choice) = v + .get("choices") + .and_then(Value::as_array) + .and_then(|a| a.first()) + { + if choice.get("finish_reason").and_then(Value::as_str) == Some("error") { + let err = choice.get("error").cloned().unwrap_or(Value::Null); + // OpenRouter's `error.code` is numeric; other OpenAI-compat hosts + // may send a string. Accept either rather than discarding the + // typed code as "unknown". + let code = err + .get("code") + .and_then(|c| { + c.as_str() + .map(str::to_string) + .or_else(|| c.as_i64().map(|n| n.to_string())) + }) + .unwrap_or_else(|| "unknown".into()); + let message = err + .get("message") + .and_then(Value::as_str) + .unwrap_or("provider error in 200 response"); + let error_type = err + .get("metadata") + .and_then(|m| m.get("error_type")) + .and_then(Value::as_str); + return Err(AgentError::Llm(match error_type { + Some(et) => format!("provider error ({code}, {et}): {message}"), + None => format!("provider error ({code}): {message}"), + })); + } + } let choice = v .get("choices") .and_then(Value::as_array) @@ -1517,9 +1615,24 @@ fn parse_openai(v: Value) -> Result { output_tokens, total_tokens, reasoning, + reasoning_details: None, }) } +fn parse_openai_with_reasoning_details(v: Value) -> Result { + let reasoning_details = v + .get("choices") + .and_then(Value::as_array) + .and_then(|a| a.first()) + .and_then(|c| c.get("message")) + .and_then(|m| m.get("reasoning_details")) + .filter(|rd| rd.is_array()) + .cloned(); + let mut response = parse_openai(v)?; + response.reasoning_details = reasoning_details; + Ok(response) +} + fn make_tool_call( id: String, name: String, @@ -1809,7 +1922,7 @@ where /// flow; subsequent requests use the cache + refresh transparently. pub(crate) fn build_token_source(cfg: &Config) -> Result, AgentError> { match cfg.provider { - Provider::Anthropic | Provider::OpenAi => { + Provider::Anthropic | Provider::OpenAi | Provider::OpenRouter => { Ok(Arc::new(StaticTokenSource::new(cfg.api_key.clone()))) } Provider::Databricks | Provider::DatabricksV2 => { @@ -1835,6 +1948,29 @@ pub(crate) fn build_token_source(cfg: &Config) -> Result, A } } +/// Build the request body for `Llm::summarize` on `Provider::OpenRouter`. +/// Extracted so tests can assert on the actual wire shape instead of a +/// hand-rolled literal — summaries never carry `reasoning` (see +/// `apply_openrouter_mutations`, which the summary path never calls). +/// It spells the token limit `max_tokens` directly for the same reason: the +/// mutation that renames it is never applied here. +fn openrouter_summary_body( + effective_model: &str, + system_prompt: &str, + user_prompt: &str, + max_output_tokens: u32, +) -> Value { + json!({ + "model": effective_model, + "stream": false, + "max_tokens": max_output_tokens, + "messages": [ + { "role": "system", "content": system_prompt }, + { "role": "user", "content": user_prompt }, + ], + }) +} + /// Return a clone of `body` with any top-level `"model"` field removed. /// Used for Databricks model-serving, which encodes the model in the URL /// path and rejects the field in the body. @@ -1849,12 +1985,353 @@ fn strip_model(body: &Value) -> Value { } } +#[derive(Debug)] +enum OpenRouterErrorClass { + Retryable(Option), + Unknown, +} + +/// Ceiling applied to the server-supplied `Retry-After` header before we +/// sleep on it. OpenRouter can advertise waits up to an hour, but +/// `openrouter_post`'s per-attempt sleep happens *outside* +/// `Client::timeout` (`cfg.llm_timeout`, default 240s) — an unclamped hint +/// could keep a single turn alive for up to two full-duration sleeps across +/// `MAX_RETRIES`. Clamping (never rejecting) keeps us honoring the server's +/// backoff signal while bounding worst-case turn latency to a value smaller +/// than the connect/response timeout. +const RETRY_AFTER_CAP_SECS: u64 = 60; + +fn parse_retry_after_header(headers: &reqwest::header::HeaderMap) -> Option { + let val = headers.get(reqwest::header::RETRY_AFTER)?.to_str().ok()?; + let secs: u64 = val.trim().parse().ok()?; + (secs > 0).then(|| std::time::Duration::from_secs(secs.min(RETRY_AFTER_CAP_SECS))) +} + +fn classify_openrouter_error( + status: u16, + body: &str, + header_retry_after: Option, +) -> OpenRouterErrorClass { + let parsed: Option = serde_json::from_str(body).ok(); + let error_type = parsed + .as_ref() + .and_then(|v| v.get("error")) + .and_then(|e| e.get("metadata")) + .and_then(|m| m.get("error_type")) + .and_then(Value::as_str); + // OpenRouter's documented retry hint is the HTTP `Retry-After` header + // (see https://openrouter.ai/docs/api_reference/errors-and-debugging); + // no current doc specifies a body-level retry field, so we don't parse one. + let retry_after = header_retry_after; + + match (status, error_type) { + (429, _) => OpenRouterErrorClass::Retryable(retry_after), + (502, _) => OpenRouterErrorClass::Retryable(None), + (503, Some("provider_overloaded")) => OpenRouterErrorClass::Retryable(retry_after), + _ => OpenRouterErrorClass::Unknown, + } +} + +/// The one place the parameter-routing failure is worded. OpenRouter reports it +/// two ways — a 404 `No endpoints found ...` (what a `require_parameters`-style +/// or unsupported-parameter body actually returns) and an untyped 503 — and both +/// mean the same thing to the user: the model id is fine, the request shape is +/// not serveable by any endpoint behind it. +fn openrouter_parameter_routing_error(error_body: &str) -> AgentError { + AgentError::Llm(format!( + "no OpenRouter endpoint supports the requested parameters — \ + check model, effort, and tool requirements: {error_body}" + )) +} + +async fn openrouter_post( + http: &Client, + url: &str, + body: &Value, + bearer: &str, +) -> Result { + let body_bytes = + serde_json::to_vec(body).map_err(|e| AgentError::Llm(format!("serialize: {e}")))?; + let call_start = std::time::Instant::now(); + for attempt in 0..MAX_RETRIES { + let resp = match http + .post(url) + .header("content-type", "application/json") + .header("HTTP-Referer", "https://github.com/block/buzz") + .header("X-OpenRouter-Title", "Buzz") + .bearer_auth(bearer) + .body(body_bytes.clone()) + .send() + .await + { + Ok(r) => r, + Err(e) => { + if attempt + 1 < MAX_RETRIES && is_retryable_transport_error(&e) { + tracing::warn!( + attempt = attempt + 1, + max_attempts = MAX_RETRIES, + error = %e, + "llm: openrouter transport error, retrying" + ); + backoff_with_jitter(attempt).await; + continue; + } + return Err(terminal_llm_error( + call_start.elapsed(), + attempt + 1, + &format!("transport: {e}"), + )); + } + }; + let status = resp.status(); + // Unlike the generic `post` path, OpenRouter's static-key auth makes + // 401 and 403 distinguishable: 401 is an invalid/expired key (worth + // one refresh-and-retry), while OpenRouter documents 403 as a + // guardrail/moderation/permission rejection the same key will always + // reproduce. Refreshing a static key returns the identical key, so + // classifying 403 as `LlmAuth` would just waste a duplicate request + // and surface Desktop's unrelated "access denied" copy. + if status == 401 { + return Err(AgentError::LlmAuth(read_error_body(resp).await)); + } + if status == 403 { + return Err(AgentError::Llm(format!( + "{status}: {}", + read_error_body(resp).await + ))); + } + if status == 402 { + return Err(AgentError::Llm( + "OpenRouter credits exhausted — check https://openrouter.ai/credits".into(), + )); + } + if status == 404 { + // OpenRouter overloads 404: a genuinely unknown/unavailable model id + // and a valid model whose parameter set no endpoint can serve both + // land here. Discriminate on the full parameter-routing phrase rather + // than a prefix — "No endpoints found for "-style bodies are + // about the model, and reporting a parameter problem as + // `LlmModelNotFound` (or vice versa) sends the user to the wrong fix. + let error_body = read_error_body(resp).await; + if error_body.contains("No endpoints found that can handle the requested parameters") { + return Err(openrouter_parameter_routing_error(&error_body)); + } + return Err(AgentError::LlmModelNotFound(format!( + "{status}: {error_body}" + ))); + } + // A6: status+error_type retry matrix + // 499 (Client Closed Request) is included: OpenRouter may emit it when a + // turn times out mid-stream. Will added 499 to the shared `post()` path in + // #2175 for the same reason; OpenRouter must match to avoid silently opting + // out of that stall-surfacing recovery. + if status.is_server_error() || status == 429 || status.as_u16() == 499 { + let header_retry_after = parse_retry_after_header(resp.headers()); + let error_body = read_error_body(resp).await; + let should_retry = if attempt + 1 < MAX_RETRIES { + match classify_openrouter_error(status.as_u16(), &error_body, header_retry_after) { + OpenRouterErrorClass::Retryable(delay) => { + if let Some(d) = delay { + tokio::time::sleep(d).await; + } else { + backoff_with_jitter(attempt).await; + } + true + } + OpenRouterErrorClass::Unknown => { + backoff_with_jitter(attempt).await; + true + } + } + } else { + false + }; + if should_retry { + continue; + } + // Terminal: classify for the user + return if status == 429 { + Err(terminal_llm_error( + call_start.elapsed(), + attempt + 1, + &format!("rate limited: {error_body}"), + )) + } else { + let parsed: Option = serde_json::from_str(&error_body).ok(); + let has_error_type = parsed + .as_ref() + .and_then(|v| v.get("error")) + .and_then(|e| e.get("metadata")) + .and_then(|m| m.get("error_type")) + .and_then(Value::as_str) + .is_some(); + Err(if !has_error_type && status.as_u16() == 503 { + openrouter_parameter_routing_error(&error_body) + } else { + terminal_llm_error( + call_start.elapsed(), + attempt + 1, + &format!("exhausted retries: {status}: {error_body}"), + ) + }) + }; + } + if !status.is_success() { + return Err(AgentError::Llm(format!( + "{status}: {}", + read_error_body(resp).await + ))); + } + if let Some(len) = resp.content_length() { + if len as usize > MAX_LLM_RESPONSE_BYTES { + return Err(AgentError::Llm(format!( + "response too large: {len} > {MAX_LLM_RESPONSE_BYTES}" + ))); + } + } + let mut buf: Vec = Vec::new(); + let mut stream = resp; + loop { + match stream.chunk().await { + Ok(Some(chunk)) => { + if buf.len() + chunk.len() > MAX_LLM_RESPONSE_BYTES { + return Err(AgentError::Llm(format!( + "response exceeded {MAX_LLM_RESPONSE_BYTES} bytes" + ))); + } + buf.extend_from_slice(&chunk); + } + Ok(None) => break, + Err(e) => { + return Err(terminal_llm_error( + call_start.elapsed(), + attempt + 1, + &format!("body read: {e}"), + )) + } + } + } + return serde_json::from_slice(&buf).map_err(|e| AgentError::Llm(format!("json: {e}"))); + } + Err(terminal_llm_error( + call_start.elapsed(), + MAX_RETRIES, + "exhausted retries", + )) +} + +fn apply_openrouter_mutations( + body: &mut Value, + effort: Option, + effective_model: &str, + prompt_caching: bool, +) { + if let Some(obj) = body.as_object_mut() { + // OpenRouter's Chat Completions API spells the output cap `max_tokens`; + // `max_completion_tokens` is OpenAI-native and only 53 of 274 + // tools-capable OpenRouter models advertise it. Sending the OpenAI + // spelling risks the cap being dropped on the floor by the endpoint we + // route to, so translate it here rather than at the shared `openai_body` + // (which OpenAI and Databricks also use, where the OpenAI spelling is + // correct). + if let Some(max_tokens) = obj.remove("max_completion_tokens") { + obj.insert("max_tokens".into(), max_tokens); + } + + // A2/A3: Add OpenRouter reasoning object when effort is configured. + // Deliberately NOT paired with `provider.require_parameters`: that filter + // routes only to endpoints advertising every parameter in the body, and + // 83 of 274 tools-capable models do not advertise `reasoning`, so it turns + // an opt-in effort setting into a hard 404 ("No endpoints found that can + // handle the requested parameters") on a model id that is perfectly + // valid. OpenRouter already best-effort routes on `tools`/`max_tokens` + // without the filter, so we accept a request served without reasoning + // over a request that cannot be served at all. + if let Some(e) = effort { + obj.insert( + "reasoning".into(), + json!({ "effort": e.openai_effort_str() }), + ); + } + + // A7: Anthropic cache_control injection for anthropic/* models. Gated on + // `prompt_caching` (`BUZZ_AGENT_PROMPT_CACHING`) for the same reason as + // the native Anthropic route: these are Anthropic-dialect breakpoints on + // an Anthropic model, so the documented kill switch must reach them too. + if prompt_caching && effective_model.starts_with("anthropic/") { + apply_anthropic_cache_control(obj); + } + } +} + +fn apply_anthropic_cache_control(body: &mut serde_json::Map) { + if let Some(messages) = body.get_mut("messages").and_then(Value::as_array_mut) { + // Cache the system message + if let Some(system_msg) = messages + .iter_mut() + .find(|m| m.get("role").and_then(Value::as_str) == Some("system")) + { + if let Some(content) = system_msg.get("content").and_then(Value::as_str) { + let content_str = content.to_string(); + if let Some(obj) = system_msg.as_object_mut() { + obj.insert( + "content".into(), + json!([{ + "type": "text", + "text": content_str, + "cache_control": { "type": "ephemeral" } + }]), + ); + } + } + } + + // Cache last 2 user messages (skip image-only ones — A7 mixed-content regression) + let mut user_count = 0; + for msg in messages.iter_mut().rev() { + if msg.get("role").and_then(Value::as_str) != Some("user") { + continue; + } + // Only cache string content (plain text user messages), not array content + // (image batches from tool results). This prevents corrupting image-only + // user messages by converting them to text cache breakpoints. + if let Some(content) = msg.get("content").and_then(Value::as_str) { + let content_str = content.to_string(); + if let Some(obj) = msg.as_object_mut() { + obj.insert( + "content".into(), + json!([{ + "type": "text", + "text": content_str, + "cache_control": { "type": "ephemeral" } + }]), + ); + } + user_count += 1; + } + if user_count >= 2 { + break; + } + } + } + // Cache the last tool definition + if let Some(tools) = body.get_mut("tools").and_then(Value::as_array_mut) { + if let Some(last_tool) = tools.last_mut() { + if let Some(function) = last_tool.get_mut("function").and_then(Value::as_object_mut) { + function.insert("cache_control".into(), json!({ "type": "ephemeral" })); + } + } + } +} + #[cfg(test)] mod tests { use super::*; use crate::config::{Config, HookServers, OpenAiApi, Provider}; use crate::types::{HistoryItem, ToolCall, ToolResult, ToolResultContent}; + use std::collections::VecDeque; use std::time::Duration; + use tokio::sync::Mutex; use tracing_subscriber::layer::SubscriberExt; fn cfg(provider: Provider) -> Config { @@ -2492,6 +2969,7 @@ mod tests { arguments: serde_json::json!({"source":"x.png"}), provider_extra: Default::default(), }], + reasoning_details: None, }, HistoryItem::ToolResult(ToolResult { provider_id: "toolu_1".into(), @@ -2542,6 +3020,7 @@ mod tests { arguments: serde_json::json!({"command": "ls"}), provider_extra: Default::default(), }], + reasoning_details: None, }, HistoryItem::ToolResult(ToolResult { provider_id: "call_abc".into(), @@ -2641,6 +3120,7 @@ mod tests { arguments: serde_json::json!({}), provider_extra: Default::default(), }], + reasoning_details: None, }, ]; let body = responses_body(&cfg_responses(), "system", &history, &[], "model", None); @@ -2927,6 +3407,7 @@ mod tests { provider_extra: Default::default(), }, ], + reasoning_details: None, }, HistoryItem::ToolResult(ToolResult { provider_id: "toolu_a".into(), @@ -2988,6 +3469,7 @@ mod tests { HistoryItem::Assistant { text: "hi".into(), tool_calls: vec![], + reasoning_details: None, }, HistoryItem::User("more".into()), ], @@ -4003,6 +4485,7 @@ mod tests { let history = vec![HistoryItem::Assistant { text: r.text.clone(), tool_calls: r.tool_calls.clone(), + reasoning_details: None, }]; let body = openai_body( &cfg(Provider::DatabricksV2), @@ -4597,4 +5080,1476 @@ mod tests { }); assert_eq!(parse_responses(v).unwrap().output_tokens, None); } + + // ---- A3: OpenRouter body-shape tests ---- + + fn tools_vec() -> Vec { + vec![ToolDef { + name: "dev__shell".into(), + description: "run a shell command".into(), + input_schema: serde_json::json!({ + "type": "object", + "properties": {"command": {"type": "string"}}, + }), + }] + } + + #[test] + fn openrouter_body_tools_with_effort() { + let mut c = cfg(Provider::OpenRouter); + c.thinking_effort = Some(ThinkingEffort::High); + let mut body = openai_body( + &c, + "system", + &[HistoryItem::User("hi".into())], + &tools_vec(), + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations( + &mut body, + c.thinking_effort, + "anthropic/claude-opus-4-7", + true, + ); + assert_eq!(body["reasoning"]["effort"], "high"); + // `openai_body` is always called with `effort=None` on the OpenRouter + // path (line 151): OpenRouter uses its own `reasoning` object, not the + // OpenAI-style `reasoning_effort` field. Verify the field is structurally + // absent — not merely removed by a no-op cleanup. + assert!( + body.get("reasoning_effort").is_none(), + "reasoning_effort must never appear on the OpenRouter body path: \ + openai_body is called with effort=None, so the field is never emitted" + ); + assert!( + body.get("provider").is_none(), + "provider.require_parameters hard-404s models that do not advertise \ + every parameter we send" + ); + assert!(!body["tools"].as_array().unwrap().is_empty()); + assert_eq!( + body["max_tokens"], 1024, + "token limit must carry openai_body's value under OpenRouter's spelling" + ); + assert!( + body.get("max_completion_tokens").is_none(), + "max_completion_tokens is the OpenAI-native spelling; OpenRouter reads max_tokens" + ); + } + + #[test] + fn openrouter_body_tools_no_effort() { + let c = cfg(Provider::OpenRouter); + let mut body = openai_body( + &c, + "system", + &[HistoryItem::User("hi".into())], + &tools_vec(), + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations(&mut body, None, "anthropic/claude-opus-4-7", true); + assert!( + body.get("reasoning").is_none(), + "reasoning must be absent when effort is None" + ); + assert!( + body.get("reasoning_effort").is_none(), + "reasoning_effort must never appear on the OpenRouter body path" + ); + assert!( + body.get("provider").is_none(), + "tools alone must not add a provider routing filter" + ); + } + + #[test] + fn openrouter_body_empty_tools_with_effort() { + let mut c = cfg(Provider::OpenRouter); + c.thinking_effort = Some(ThinkingEffort::Medium); + let mut body = openai_body( + &c, + "system", + &[HistoryItem::User("hi".into())], + &[], + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations( + &mut body, + c.thinking_effort, + "anthropic/claude-opus-4-7", + true, + ); + assert_eq!(body["reasoning"]["effort"], "medium"); + assert!( + body.get("provider").is_none(), + "83 of 274 tools-capable models do not advertise `reasoning`; filtering on \ + it would 404 them instead of answering without reasoning" + ); + } + + #[test] + fn openrouter_body_empty_tools_no_effort() { + let c = cfg(Provider::OpenRouter); + let mut body = openai_body( + &c, + "system", + &[HistoryItem::User("hi".into())], + &[], + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations(&mut body, None, "anthropic/claude-opus-4-7", true); + assert!(body.get("reasoning").is_none()); + assert!( + body.get("provider").is_none(), + "no body shape adds a provider routing filter" + ); + } + + /// `BUZZ_AGENT_PROMPT_CACHING=0` must reach the OpenRouter `anthropic/*` + /// route too, not just the native Anthropic Messages routes. The switch and + /// this route landed in separate changes, so nothing but this test stops the + /// gate from being dropped and the kill switch silently becoming a no-op on + /// a route that emits Anthropic-dialect breakpoints. + #[test] + fn openrouter_body_caching_disabled_emits_no_cache_control() { + let c = cfg(Provider::OpenRouter); + let mut body = openai_body( + &c, + "system", + &[HistoryItem::User("hi".into())], + &tools_vec(), + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations(&mut body, None, "anthropic/claude-opus-4-7", false); + assert!( + !body.to_string().contains("cache_control"), + "BUZZ_AGENT_PROMPT_CACHING=0 must suppress every breakpoint: {body}" + ); + // The switch is scoped to caching — the routing-contract mutations that + // make the request serveable at all must still be applied. + assert_eq!( + body.get("max_tokens").and_then(Value::as_u64), + Some(u64::from(c.max_output_tokens)), + "the max_tokens rename is not part of the caching gate" + ); + } + + /// The paired positive case: with caching on, the same body does carry + /// breakpoints. Without this twin, a mutation that hard-disabled caching + /// outright would still leave the test above green. + #[test] + fn openrouter_body_caching_enabled_emits_cache_control() { + let c = cfg(Provider::OpenRouter); + let mut body = openai_body( + &c, + "system", + &[HistoryItem::User("hi".into())], + &tools_vec(), + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations(&mut body, None, "anthropic/claude-opus-4-7", true); + assert!( + body.to_string().contains("cache_control"), + "caching on must still emit breakpoints: {body}" + ); + } + + /// The rename moves an existing value; it never invents a token limit for a + /// body that did not carry one. + #[test] + fn openrouter_body_without_token_limit_gains_none() { + let mut body = json!({ "model": "vendor/model", "messages": [] }); + apply_openrouter_mutations(&mut body, None, "vendor/model", true); + assert!(body.get("max_tokens").is_none()); + assert!(body.get("max_completion_tokens").is_none()); + } + + #[test] + fn openrouter_summary_carries_neither_reasoning_nor_provider() { + let body = openrouter_summary_body( + "anthropic/claude-opus-4-7", + "summarize", + "text to summarize", + 1024, + ); + assert_eq!(body["model"], "anthropic/claude-opus-4-7"); + assert_eq!(body["messages"][0]["role"], "system"); + assert_eq!(body["messages"][1]["content"], "text to summarize"); + assert_eq!(body["max_tokens"], 1024); + assert!( + body.get("max_completion_tokens").is_none(), + "summary body must use OpenRouter's token-limit spelling" + ); + assert!( + body.get("reasoning").is_none(), + "summary body must not carry reasoning" + ); + assert!( + body.get("provider").is_none(), + "summary body must not carry provider" + ); + } + + /// The token-limit rename belongs to `apply_openrouter_mutations`, not to + /// `openai_body` — which OpenAI and Databricks also use, and where + /// `max_completion_tokens` is the correct spelling. + #[test] + fn openai_body_keeps_max_completion_tokens_when_unmutated() { + let body = openai_body( + &cfg(Provider::OpenAi), + "system", + &[HistoryItem::User("hi".into())], + &[], + "model", + None, + ); + assert_eq!(body["max_completion_tokens"], 1024); + assert!(body.get("max_tokens").is_none()); + } + + // ---- A5: error-inside-200 ---- + + #[test] + fn parse_openai_error_inside_200_returns_error() { + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "error", + "error": { + "code": 503, + "message": "No endpoints found that support tool use" + } + }] + }); + let err = parse_openai(v).unwrap_err(); + match &err { + AgentError::Llm(s) => { + assert!(s.contains("provider error (503)"), "got: {s}"); + assert!(s.contains("No endpoints found"), "got: {s}"); + } + _ => panic!("expected AgentError::Llm, got: {err:?}"), + } + } + + #[test] + fn parse_openai_error_inside_200_accepts_string_code() { + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "error", + "error": { + "code": "insufficient_quota", + "message": "quota exceeded" + } + }] + }); + let err = parse_openai(v).unwrap_err(); + match &err { + AgentError::Llm(s) => assert!( + s.contains("provider error (insufficient_quota)"), + "got: {s}" + ), + _ => panic!("expected AgentError::Llm, got: {err:?}"), + } + } + + #[test] + fn parse_openai_error_inside_200_surfaces_error_type() { + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "error", + "error": { + "code": 429, + "message": "Rate limit exceeded", + "metadata": { "error_type": "rate_limit_exceeded" } + } + }] + }); + let err = parse_openai(v).unwrap_err(); + match &err { + AgentError::Llm(s) => { + assert!(s.contains("429"), "got: {s}"); + assert!(s.contains("rate_limit_exceeded"), "got: {s}"); + } + _ => panic!("expected AgentError::Llm, got: {err:?}"), + } + } + + #[test] + fn parse_openai_normal_stop_not_affected_by_error_check() { + let v = serde_json::json!({ + "choices": [{"finish_reason": "stop", "message": {"content": "hello"}}], + "usage": {"prompt_tokens": 10, "completion_tokens": 5} + }); + let r = parse_openai(v).unwrap(); + assert_eq!(r.text, "hello"); + assert_eq!(r.stop, ProviderStop::EndTurn); + } + + #[test] + fn parse_openai_tool_calls_not_affected_by_error_check() { + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "tool_calls", + "message": { + "content": "", + "tool_calls": [{ + "id": "call_1", + "type": "function", + "function": {"name": "test", "arguments": "{}"} + }] + } + }] + }); + let r = parse_openai(v).unwrap(); + assert_eq!(r.stop, ProviderStop::ToolUse); + assert_eq!(r.tool_calls.len(), 1); + } + + // ---- A6: OpenRouter retry classification ---- + + #[test] + fn classify_429_rate_limit_body_retry_after_is_ignored() { + // M1: no documented body-level retry field, so a `retry_after` key + // inside `error.metadata` is inert — only the HTTP header (passed + // separately) can produce a delay. + let body = + r#"{"error":{"metadata":{"error_type":"rate_limit_exceeded","retry_after":2.5}}}"#; + match classify_openrouter_error(429, body, None) { + OpenRouterErrorClass::Retryable(None) => {} + other => panic!("expected Retryable(None), got: {other:?}"), + } + } + + #[test] + fn classify_429_rate_limit_without_retry_after() { + let body = r#"{"error":{"metadata":{"error_type":"rate_limit_exceeded"}}}"#; + match classify_openrouter_error(429, body, None) { + OpenRouterErrorClass::Retryable(None) => {} + other => panic!("expected Retryable(None), got: {other:?}"), + } + } + + #[test] + fn classify_429_prefers_http_header_over_body() { + // Body-level retry hints are no longer parsed (M1: undocumented field); + // the HTTP header is the only source, and it's honored when present. + let body = r#"{"error":{"metadata":{"error_type":"rate_limit_exceeded"}}}"#; + let header = Some(Duration::from_secs(3)); + match classify_openrouter_error(429, body, header) { + OpenRouterErrorClass::Retryable(Some(d)) => { + assert_eq!(d, Duration::from_secs(3), "HTTP header must be honored"); + } + other => panic!("expected Retryable with header delay, got: {other:?}"), + } + } + + #[test] + fn classify_502_provider_unavailable() { + let body = r#"{"error":{"metadata":{"error_type":"provider_unavailable"}}}"#; + match classify_openrouter_error(502, body, None) { + OpenRouterErrorClass::Retryable(None) => {} + other => panic!("expected Retryable(None) for 502, got: {other:?}"), + } + } + + #[test] + fn classify_503_provider_overloaded_with_retry_after() { + // No documented body-level retry field (M1); the header is the only + // source `classify_openrouter_error` consults. + let body = r#"{"error":{"metadata":{"error_type":"provider_overloaded"}}}"#; + let header = Some(Duration::from_secs(5)); + match classify_openrouter_error(503, body, header) { + OpenRouterErrorClass::Retryable(Some(d)) => { + assert_eq!(d, Duration::from_secs(5)); + } + other => panic!("expected Retryable with delay, got: {other:?}"), + } + } + + #[test] + fn classify_503_untyped_is_unknown() { + let body = r#"{"error":{"message":"No endpoints found"}}"#; + match classify_openrouter_error(503, body, None) { + OpenRouterErrorClass::Unknown => {} + other => panic!("expected Unknown for untyped 503, got: {other:?}"), + } + } + + #[test] + fn classify_500_untyped_is_unknown() { + match classify_openrouter_error(500, r#"{"error":{"message":"internal"}}"#, None) { + OpenRouterErrorClass::Unknown => {} + other => panic!("expected Unknown for 500, got: {other:?}"), + } + } + + #[test] + fn parse_retry_after_header_valid() { + let mut headers = reqwest::header::HeaderMap::new(); + headers.insert(reqwest::header::RETRY_AFTER, "5".parse().unwrap()); + assert_eq!( + parse_retry_after_header(&headers), + Some(Duration::from_secs(5)) + ); + } + + #[test] + fn parse_retry_after_header_zero_rejected() { + let mut headers = reqwest::header::HeaderMap::new(); + headers.insert(reqwest::header::RETRY_AFTER, "0".parse().unwrap()); + assert_eq!(parse_retry_after_header(&headers), None); + } + + #[test] + fn parse_retry_after_header_over_cap_clamped() { + let mut headers = reqwest::header::HeaderMap::new(); + headers.insert(reqwest::header::RETRY_AFTER, "3601".parse().unwrap()); + assert_eq!( + parse_retry_after_header(&headers), + Some(Duration::from_secs(RETRY_AFTER_CAP_SECS)), + "over-cap hints clamp to the ceiling rather than being dropped" + ); + } + + #[test] + fn parse_retry_after_header_at_cap_unclamped() { + let mut headers = reqwest::header::HeaderMap::new(); + headers.insert( + reqwest::header::RETRY_AFTER, + RETRY_AFTER_CAP_SECS.to_string().parse().unwrap(), + ); + assert_eq!( + parse_retry_after_header(&headers), + Some(Duration::from_secs(RETRY_AFTER_CAP_SECS)) + ); + } + + #[test] + fn parse_retry_after_header_missing() { + let headers = reqwest::header::HeaderMap::new(); + assert_eq!(parse_retry_after_header(&headers), None); + } + + #[test] + fn parse_retry_after_header_non_numeric_ignored() { + let mut headers = reqwest::header::HeaderMap::new(); + headers.insert( + reqwest::header::RETRY_AFTER, + "Wed, 21 Oct 2026 07:28:00 GMT".parse().unwrap(), + ); + assert_eq!(parse_retry_after_header(&headers), None); + } + + // ---- A7: Anthropic cache_control with mixed content ---- + + #[test] + fn anthropic_cache_control_mixed_text_tool_image_history() { + let history = vec![ + HistoryItem::User("first question".into()), + HistoryItem::Assistant { + text: String::new(), + tool_calls: vec![ToolCall { + provider_id: "toolu_1".into(), + name: "dev__view_image".into(), + arguments: serde_json::json!({"source": "x.png"}), + provider_extra: Default::default(), + }], + reasoning_details: None, + }, + HistoryItem::ToolResult(ToolResult { + provider_id: "toolu_1".into(), + content: vec![ + ToolResultContent::Text("10×10 image".into()), + ToolResultContent::Image { + data: "aW1n".into(), + mime_type: "image/png".into(), + }, + ], + is_error: false, + }), + HistoryItem::User("second question about the image".into()), + HistoryItem::User("third question".into()), + ]; + let mut body = openai_body( + &cfg(Provider::OpenRouter), + "system", + &history, + &tools_vec(), + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations(&mut body, None, "anthropic/claude-opus-4-7", true); + let messages = body["messages"].as_array().unwrap(); + + // System message should have cache_control + let system = &messages[0]; + assert_eq!(system["content"][0]["cache_control"]["type"], "ephemeral"); + + // Image-batch user messages (containing image_url blocks) must NOT have cache_control + let image_user_msgs: Vec<_> = messages + .iter() + .filter(|m| { + m.get("role").and_then(Value::as_str) == Some("user") + && m.get("content") + .and_then(Value::as_array) + .map(|a| { + a.iter() + .any(|b| b.get("type").and_then(Value::as_str) == Some("image_url")) + }) + .unwrap_or(false) + }) + .collect(); + assert!( + !image_user_msgs.is_empty(), + "should have image user messages" + ); + for img_msg in &image_user_msgs { + let content = img_msg["content"].as_array().unwrap(); + for block in content { + assert!( + block.get("cache_control").is_none(), + "image-only user message must not receive cache_control" + ); + } + } + + // Exactly 2 text user messages should have cache_control (skipping image-only ones) + let cached_text_count = messages + .iter() + .filter(|m| { + m.get("role").and_then(Value::as_str) == Some("user") + && m.get("content") + .and_then(Value::as_array) + .map(|a| { + a.iter().any(|b| { + b.get("type").and_then(Value::as_str) == Some("text") + && b.get("cache_control").is_some() + }) + }) + .unwrap_or(false) + }) + .count(); + assert_eq!( + cached_text_count, 2, + "exactly 2 text user messages should have cache_control" + ); + + // Last tool def should have cache_control + let tools = body["tools"].as_array().unwrap(); + let last_tool = tools.last().unwrap(); + assert_eq!(last_tool["function"]["cache_control"]["type"], "ephemeral"); + } + + #[test] + fn anthropic_cache_control_image_only_user_does_not_consume_slot() { + // An image-only user message between two text user messages must not + // consume a cache breakpoint slot — both text messages should get cached. + let history = vec![ + HistoryItem::User("text message one".into()), + HistoryItem::Assistant { + text: String::new(), + tool_calls: vec![ToolCall { + provider_id: "toolu_1".into(), + name: "dev__view_image".into(), + arguments: serde_json::json!({"source": "x.png"}), + provider_extra: Default::default(), + }], + reasoning_details: None, + }, + HistoryItem::ToolResult(ToolResult { + provider_id: "toolu_1".into(), + content: vec![ToolResultContent::Image { + data: "aW1n".into(), + mime_type: "image/png".into(), + }], + is_error: false, + }), + HistoryItem::User("text message two".into()), + HistoryItem::User("text message three".into()), + ]; + let mut body = openai_body( + &cfg(Provider::OpenRouter), + "system", + &history, + &[], + "anthropic/claude-opus-4-7", + None, + ); + apply_openrouter_mutations(&mut body, None, "anthropic/claude-opus-4-7", true); + let messages = body["messages"].as_array().unwrap(); + + // Count text user messages that got cache_control + let cached_text_count = messages + .iter() + .filter(|m| { + m.get("role").and_then(Value::as_str) == Some("user") + && m.get("content") + .and_then(Value::as_array) + .map(|a| a.iter().any(|b| b.get("cache_control").is_some())) + .unwrap_or(false) + }) + .count(); + assert_eq!( + cached_text_count, 2, + "image-only user messages must not consume a cache breakpoint slot" + ); + } + + // ---- A9: reasoning_details round-trip ---- + + #[test] + fn parse_openai_with_reasoning_details_captures_array() { + let details = serde_json::json!([ + {"type": "thinking", "content": "Let me consider..."}, + {"type": "thinking", "content": "The answer is 42."} + ]); + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "tool_calls", + "message": { + "content": "", + "reasoning_details": details, + "tool_calls": [{ + "id": "call_1", + "type": "function", + "function": {"name": "test", "arguments": "{}"} + }] + } + }], + "usage": {"prompt_tokens": 10, "completion_tokens": 5} + }); + let r = parse_openai_with_reasoning_details(v).unwrap(); + assert_eq!(r.reasoning_details, Some(details)); + } + + #[test] + fn parse_openai_with_reasoning_details_none_when_absent() { + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "stop", + "message": {"content": "hello"} + }] + }); + let r = parse_openai_with_reasoning_details(v).unwrap(); + assert!( + r.reasoning_details.is_none(), + "reasoning_details must be None when not in response" + ); + } + + /// M2: a malformed `reasoning_details` shape (null or a bare object, + /// rather than the documented array) must be omitted at the parse + /// boundary, not stored and later replayed into the next request. + #[test] + fn parse_openai_with_reasoning_details_omits_non_array_shapes() { + let null_shape = serde_json::json!({ + "choices": [{ + "finish_reason": "stop", + "message": {"content": "hello", "reasoning_details": null} + }] + }); + let r = parse_openai_with_reasoning_details(null_shape).unwrap(); + assert!( + r.reasoning_details.is_none(), + "null reasoning_details must be omitted, not stored as Some(Null)" + ); + + let object_shape = serde_json::json!({ + "choices": [{ + "finish_reason": "stop", + "message": { + "content": "hello", + "reasoning_details": {"type": "thinking", "content": "not an array"} + } + }] + }); + let r = parse_openai_with_reasoning_details(object_shape).unwrap(); + assert!( + r.reasoning_details.is_none(), + "a bare-object reasoning_details must be omitted, not stored as Some(object)" + ); + } + + #[test] + fn parse_openai_plain_never_captures_reasoning_details() { + let v = serde_json::json!({ + "choices": [{ + "finish_reason": "stop", + "message": { + "content": "hello", + "reasoning_details": [{"type": "thinking", "content": "hmm"}] + } + }] + }); + let r = parse_openai(v).unwrap(); + assert!( + r.reasoning_details.is_none(), + "plain parse_openai must never capture reasoning_details (OpenAI/Databricks regression)" + ); + } + + #[test] + fn reasoning_details_two_request_round_trip() { + let details = serde_json::json!([ + {"type": "thinking", "content": "Step 1: analyze the request."}, + {"type": "thinking", "content": "Step 2: call the tool."} + ]); + // Request 1: model returns a tool call with reasoning_details + let response1 = serde_json::json!({ + "choices": [{ + "finish_reason": "tool_calls", + "message": { + "content": "", + "reasoning_details": details, + "tool_calls": [{ + "id": "call_abc", + "type": "function", + "function": {"name": "dev__shell", "arguments": "{\"command\":\"ls\"}"} + }] + } + }], + "usage": {"prompt_tokens": 50, "completion_tokens": 20} + }); + let r1 = parse_openai_with_reasoning_details(response1).unwrap(); + assert_eq!(r1.reasoning_details, Some(details.clone())); + + // Build history as the agent would: assistant turn with reasoning_details, + // followed by a tool result. + let history = vec![ + HistoryItem::User("run ls".into()), + HistoryItem::Assistant { + text: String::new(), + tool_calls: r1.tool_calls, + reasoning_details: r1.reasoning_details, + }, + HistoryItem::ToolResult(ToolResult { + provider_id: "call_abc".into(), + content: vec![ToolResultContent::Text("file.txt".into())], + is_error: false, + }), + ]; + + // Request 2: build the body for the continuation + let body = openai_body( + &cfg(Provider::OpenRouter), + "system", + &history, + &[], + "anthropic/claude-opus-4-7", + None, + ); + let messages = body["messages"].as_array().unwrap(); + + // The assistant message must carry the identical reasoning_details array + let assistant_msg = messages + .iter() + .find(|m| m.get("role").and_then(Value::as_str) == Some("assistant")) + .expect("assistant message must exist"); + assert_eq!( + assistant_msg["reasoning_details"], details, + "reasoning_details must be replayed byte-for-byte on the assistant message" + ); + + // The assistant message must appear BEFORE the tool result + let assistant_idx = messages + .iter() + .position(|m| m.get("role").and_then(Value::as_str) == Some("assistant")) + .unwrap(); + let tool_idx = messages + .iter() + .position(|m| m.get("role").and_then(Value::as_str) == Some("tool")) + .unwrap(); + assert!( + assistant_idx < tool_idx, + "assistant with reasoning_details must precede tool result" + ); + } + + #[test] + fn reasoning_details_none_emits_no_field_in_body() { + let history = vec![ + HistoryItem::User("hello".into()), + HistoryItem::Assistant { + text: "hi back".into(), + tool_calls: Vec::new(), + reasoning_details: None, + }, + ]; + let body = openai_body( + &cfg(Provider::OpenRouter), + "system", + &history, + &[], + "anthropic/claude-opus-4-7", + None, + ); + let messages = body["messages"].as_array().unwrap(); + let assistant_msg = messages + .iter() + .find(|m| m.get("role").and_then(Value::as_str) == Some("assistant")) + .expect("assistant message must exist"); + assert!( + assistant_msg.get("reasoning_details").is_none(), + "assistant with None reasoning_details must not emit the field" + ); + } + + #[test] + fn reasoning_details_charged_to_estimated_bytes() { + let details = serde_json::json!([ + {"type": "thinking", "content": "A long chain of reasoning tokens here."} + ]); + let with = HistoryItem::Assistant { + text: "text".into(), + tool_calls: Vec::new(), + reasoning_details: Some(details.clone()), + }; + let without = HistoryItem::Assistant { + text: "text".into(), + tool_calls: Vec::new(), + reasoning_details: None, + }; + assert!( + with.estimated_bytes() > without.estimated_bytes(), + "reasoning_details must contribute to estimated_bytes" + ); + assert!( + with.context_pressure_bytes() > without.context_pressure_bytes(), + "reasoning_details must contribute to context_pressure_bytes" + ); + let details_size = serde_json::to_vec(&details).unwrap().len(); + assert_eq!( + with.estimated_bytes() - without.estimated_bytes(), + details_size, + "reasoning_details contribution must equal its serialized size" + ); + } + + #[test] + fn reasoning_details_not_replayed_in_anthropic_body() { + let history = vec![ + HistoryItem::User("hi".into()), + HistoryItem::Assistant { + text: "ok".into(), + tool_calls: Vec::new(), + reasoning_details: Some( + serde_json::json!([{"type": "thinking", "content": "hmm"}]), + ), + }, + ]; + let body = anthropic_body( + &cfg(Provider::Anthropic), + "system", + &history, + &[], + "claude-opus-4-7", + None, + ); + let messages = body["messages"].as_array().unwrap(); + let assistant = messages + .iter() + .find(|m| m.get("role").and_then(Value::as_str) == Some("assistant")) + .unwrap(); + assert!( + assistant.get("reasoning_details").is_none(), + "anthropic_body must not replay reasoning_details" + ); + } + + #[test] + fn reasoning_details_not_replayed_in_responses_body() { + let history = vec![ + HistoryItem::User("hi".into()), + HistoryItem::Assistant { + text: "ok".into(), + tool_calls: Vec::new(), + reasoning_details: Some( + serde_json::json!([{"type": "thinking", "content": "hmm"}]), + ), + }, + ]; + let body = responses_body(&cfg_responses(), "system", &history, &[], "model", None); + let body_str = serde_json::to_string(&body).unwrap(); + assert!( + !body_str.contains("reasoning_details"), + "responses_body must not replay reasoning_details" + ); + } + + // ---- T4: openrouter_post transport-level regressions ---- + // + // These stub an HTTP server directly and drive `openrouter_post` (not the + // classifier in isolation), proving the retry/attempt-accounting and + // header behavior the classifier-only tests above cannot see. + + /// One canned response: status, body, and any extra headers (e.g. + /// `Retry-After`) to send back for a single request. + struct CannedResponse { + status: u16, + body: String, + extra_headers: Vec<(String, String)>, + } + + impl CannedResponse { + fn new(status: u16, body: &str) -> Self { + Self { + status, + body: body.into(), + extra_headers: Vec::new(), + } + } + + fn with_header(mut self, name: &str, value: &str) -> Self { + self.extra_headers.push((name.into(), value.into())); + self + } + } + + fn status_line(status: u16) -> &'static str { + match status { + 200 => "200 OK", + 401 => "401 Unauthorized", + 402 => "402 Payment Required", + 403 => "403 Forbidden", + 404 => "404 Not Found", + 429 => "429 Too Many Requests", + 499 => "499 Client Closed Request", + 500 => "500 Internal Server Error", + 502 => "502 Bad Gateway", + 503 => "503 Service Unavailable", + _ => panic!("unsupported status {status} in test stub"), + } + } + + /// Spawns a stub HTTP server that pops one `CannedResponse` per request + /// (repeating the last one once the queue is exhausted, so an + /// over-budget attempt count is visible rather than hanging), and + /// captures each request's raw header block for header-attribution + /// assertions. Returns (url, captured_header_blocks, attempt_counter). + async fn spawn_openrouter_stub( + responses: Vec, + ) -> ( + String, + Arc>>, + Arc, + ) { + use std::sync::atomic::{AtomicU32, Ordering}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::TcpListener; + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let url = format!("http://{}", listener.local_addr().unwrap()); + let queue = Arc::new(Mutex::new(VecDeque::from(responses))); + let captured: Arc>> = Arc::new(Mutex::new(Vec::new())); + let attempts = Arc::new(AtomicU32::new(0)); + let captured_clone = captured.clone(); + let attempts_clone = attempts.clone(); + tokio::spawn(async move { + loop { + let (mut sock, _) = match listener.accept().await { + Ok(p) => p, + Err(_) => return, + }; + let queue = queue.clone(); + let captured = captured_clone.clone(); + let attempts = attempts_clone.clone(); + tokio::spawn(async move { + let mut buf = Vec::new(); + let mut tmp = [0u8; 4096]; + while !buf.windows(4).any(|w| w == b"\r\n\r\n") { + match sock.read(&mut tmp).await { + Ok(0) | Err(_) => return, + Ok(n) => buf.extend_from_slice(&tmp[..n]), + } + if buf.len() > 1_000_000 { + return; + } + } + let header_end = buf.windows(4).position(|w| w == b"\r\n\r\n").unwrap() + 4; + let header_str = String::from_utf8_lossy(&buf[..header_end]).into_owned(); + // Drain any remaining body per Content-Length so `Connection: + // close` doesn't race the client's write. + let content_length: usize = header_str + .lines() + .find_map(|line| { + line.to_ascii_lowercase() + .strip_prefix("content-length:") + .and_then(|v| v.trim().parse().ok()) + }) + .unwrap_or(0); + let mut body_len = buf.len() - header_end; + while body_len < content_length { + match sock.read(&mut tmp).await { + Ok(0) | Err(_) => break, + Ok(n) => body_len += n, + } + } + captured.lock().await.push(header_str); + attempts.fetch_add(1, Ordering::SeqCst); + + let mut q = queue.lock().await; + let canned = if q.len() > 1 { + q.pop_front().unwrap() + } else { + // Repeat the final canned response so a test bug that + // over-retries produces a visible extra attempt + // instead of a hung connection. + let last = q.front().unwrap(); + CannedResponse { + status: last.status, + body: last.body.clone(), + extra_headers: last.extra_headers.clone(), + } + }; + drop(q); + + let mut resp = format!( + "HTTP/1.1 {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\n", + status_line(canned.status), + canned.body.len() + ); + for (name, value) in &canned.extra_headers { + resp.push_str(&format!("{name}: {value}\r\n")); + } + resp.push_str("Connection: close\r\n\r\n"); + resp.push_str(&canned.body); + let _ = sock.write_all(resp.as_bytes()).await; + let _ = sock.shutdown().await; + }); + } + }); + (url, captured, attempts) + } + + /// A 403 (guardrail/moderation/permission rejection, per OpenRouter docs) + /// must NOT be classified as `LlmAuth`: refreshing a static key returns + /// the identical key, so retrying would just waste a duplicate request. + /// Exactly one attempt, plain `AgentError::Llm` with the body preserved. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_403_single_attempt_not_auth_error() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![CannedResponse::new( + 403, + r#"{"error":{"message":"model flagged by moderation"}}"#, + )]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(5)) + .build() + .unwrap(); + let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .unwrap_err(); + assert!( + matches!(&err, AgentError::Llm(s) if s.contains("403") && s.contains("model flagged by moderation")), + "403 must surface as AgentError::Llm with status+body, not LlmAuth: got {err:?}" + ); + assert_eq!( + attempts.load(std::sync::atomic::Ordering::SeqCst), + 1, + "403 must not be retried (a refreshed static key is identical)" + ); + } + + /// A 402 short-circuits on the first attempt: no retry, one request, + /// the actionable credits-exhausted message. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_402_single_attempt_short_circuit() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![CannedResponse::new( + 402, + r#"{"error":{"message":"payment required"}}"#, + )]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(5)) + .build() + .unwrap(); + let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .unwrap_err(); + assert!( + matches!(&err, AgentError::Llm(s) if s.contains("credits exhausted")), + "got {err:?}" + ); + assert_eq!( + attempts.load(std::sync::atomic::Ordering::SeqCst), + 1, + "402 must not be retried" + ); + } + + /// A 404 whose body is OpenRouter's parameter-routing rejection is NOT a + /// missing model: the id is valid and no endpoint behind it can serve the + /// request shape. It must surface the actionable routing message rather than + /// `LlmModelNotFound`, which sends the user hunting a model-name typo. + /// Body text is the one OpenRouter actually returned in the live probe run. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_404_no_endpoints_found_is_parameter_routing_error() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![CannedResponse::new( + 404, + r#"{"error":{"message":"No endpoints found that can handle the requested parameters. To learn more about provider routing, visit: https://openrouter.ai/docs/guides/routing/provider-selection","code":404}}"#, + )]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(5)) + .build() + .unwrap(); + let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .unwrap_err(); + assert!( + matches!(&err, AgentError::Llm(s) if s.contains("no OpenRouter endpoint supports")), + "parameter-routing 404 must not be reported as a missing model: got {err:?}" + ); + assert_eq!( + attempts.load(std::sync::atomic::Ordering::SeqCst), + 1, + "404 must not be retried" + ); + } + + /// Every other 404 still maps to `LlmModelNotFound`, including one that + /// shares the `No endpoints found` prefix but is about the model rather than + /// the parameters — the discriminator is narrow enough that a genuinely + /// unavailable model keeps its own error kind (Desktop renders + /// model-not-found differently from a generic LLM failure). + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_404_unknown_model_stays_model_not_found() { + let (url, _captured, _attempts) = spawn_openrouter_stub(vec![CannedResponse::new( + 404, + r#"{"error":{"message":"No endpoints found for vendor/nonexistent-model.","code":404}}"#, + )]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(5)) + .build() + .unwrap(); + let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .unwrap_err(); + assert!( + matches!(&err, AgentError::LlmModelNotFound(s) if s.contains("404") && s.contains("vendor/nonexistent-model")), + "a model-level 404 must stay LlmModelNotFound: got {err:?}" + ); + } + + /// A 429 with `Retry-After: 1` sleeps for that duration before the retry + /// succeeds — proving the header value is actually honored, not just + /// classified. + #[tokio::test(flavor = "current_thread")] + async fn openrouter_post_429_honors_retry_after_header() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![ + CannedResponse::new(429, r#"{"error":{"message":"rate limited"}}"#) + .with_header("Retry-After", "1"), + CannedResponse::new(200, r#"{"choices":[{"message":{"content":"ok"}}]}"#), + ]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(30)) + .build() + .unwrap(); + let before = std::time::Instant::now(); + let out = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .expect("second attempt succeeds"); + assert_eq!(out["choices"][0]["message"]["content"], "ok"); + assert!( + before.elapsed() >= Duration::from_secs(1), + "must sleep at least the Retry-After hint" + ); + assert_eq!(attempts.load(std::sync::atomic::Ordering::SeqCst), 2); + } + + /// A `Retry-After` far beyond `RETRY_AFTER_CAP_SECS` must not stall the + /// retry loop for anywhere near its advertised duration — proving the + /// cap is enforced end-to-end in `openrouter_post`'s actual sleep, not + /// merely in the isolated `parse_retry_after_header` unit tests above. + /// Runs on a paused clock so a real 999999s wait would hang the test + /// instead of silently passing. + #[tokio::test(start_paused = true)] + async fn openrouter_post_429_retry_sleep_capped_despite_huge_retry_after() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![ + CannedResponse::new(429, r#"{"error":{"message":"rate limited"}}"#) + .with_header("Retry-After", "999999"), + CannedResponse::new(200, r#"{"choices":[{"message":{"content":"ok"}}]}"#), + ]) + .await; + let http = Client::builder().build().unwrap(); + let before = tokio::time::Instant::now(); + let out = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .expect("second attempt succeeds"); + assert_eq!(out["choices"][0]["message"]["content"], "ok"); + assert!( + before.elapsed() <= Duration::from_secs(RETRY_AFTER_CAP_SECS + 5), + "retry sleep must be clamped to RETRY_AFTER_CAP_SECS ({RETRY_AFTER_CAP_SECS}s), \ + not the header's 999999s: elapsed {:?}", + before.elapsed() + ); + assert_eq!(attempts.load(std::sync::atomic::Ordering::SeqCst), 2); + } + + /// An untyped 503 (no `error.metadata.error_type`) exhausts all + /// `MAX_RETRIES` attempts, then returns the actionable routing message — + /// proving attempt accounting terminates rather than retrying forever. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_untyped_503_exhausts_retries_into_actionable_message() { + let canned = CannedResponse::new(503, r#"{"error":{"message":"no capacity"}}"#); + let (url, _captured, attempts) = spawn_openrouter_stub(vec![canned]).await; + let http = Client::builder() + .timeout(Duration::from_secs(30)) + .build() + .unwrap(); + let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .unwrap_err(); + assert!( + matches!(&err, AgentError::Llm(s) if s.contains("no OpenRouter endpoint supports")), + "got {err:?}" + ); + assert_eq!( + attempts.load(std::sync::atomic::Ordering::SeqCst), + MAX_RETRIES, + "must exhaust exactly MAX_RETRIES attempts, no more" + ); + } + + /// Attribution headers (`HTTP-Referer`, `X-OpenRouter-Title`) are on the + /// actual wire request, not merely asserted against a body fixture. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_sends_attribution_headers() { + let (url, captured, _attempts) = spawn_openrouter_stub(vec![CannedResponse::new( + 200, + r#"{"choices":[{"message":{"content":"ok"}}]}"#, + )]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(5)) + .build() + .unwrap(); + openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .expect("200 succeeds"); + let headers = captured.lock().await; + let header_str = headers + .first() + .expect("one request captured") + .to_lowercase(); + assert!( + header_str.contains("http-referer: https://github.com/block/buzz"), + "got: {header_str}" + ); + assert!( + header_str.contains("x-openrouter-title: buzz"), + "got: {header_str}" + ); + } + + /// A 499 response is retried and the call succeeds on the second attempt, + /// mirroring the shared `post()` path (#2175). + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_retries_499_then_succeeds() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![ + CannedResponse::new(499, ""), + CannedResponse::new(200, r#"{"choices":[{"message":{"content":"ok"}}]}"#), + ]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(30)) + .build() + .unwrap(); + let out = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .expect("retry after 499 should succeed"); + assert_eq!(out["choices"][0]["message"]["content"], "ok"); + assert_eq!( + attempts.load(std::sync::atomic::Ordering::SeqCst), + 2, + "exactly one 499 retry" + ); + } + + /// A 502 without `provider_unavailable` still retries (the classifier's + /// unconditional-retry branch), then succeeds on attempt 2 — proving the + /// generic 502 path isn't accidentally routed to `Unknown`/terminal. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_502_retries_then_succeeds() { + let (url, _captured, attempts) = spawn_openrouter_stub(vec![ + CannedResponse::new(502, r#"{"error":{"message":"bad gateway"}}"#), + CannedResponse::new(200, r#"{"choices":[{"message":{"content":"ok"}}]}"#), + ]) + .await; + let http = Client::builder() + .timeout(Duration::from_secs(30)) + .build() + .unwrap(); + let out = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .expect("retry succeeds"); + assert_eq!(out["choices"][0]["message"]["content"], "ok"); + assert_eq!(attempts.load(std::sync::atomic::Ordering::SeqCst), 2); + } + + /// A 200 response whose body is truncated mid-stream (connection closed + /// before the full Content-Length is delivered) must surface the error + /// through `terminal_llm_error`, not a bare `AgentError::Llm("read: …")` — + /// the caller needs cumulative duration and attempt-count context to diagnose + /// an upstream that silently drops connections. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn openrouter_post_truncated_200_body_wraps_in_terminal_error() { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::TcpListener; + + // Spawn a stub that sends a 200 with Content-Length > actual body, + // then closes the connection — reqwest sees a truncated stream. + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let url = format!("http://{}", listener.local_addr().unwrap()); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 4096]; + // Read until end-of-headers, ignore body + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => break, + Ok(n) => { + if buf[..n].windows(4).any(|w| w == b"\r\n\r\n") { + break; + } + } + } + } + // Claim 1 MB of body, send only 10 bytes, then close. + let _ = sock + .write_all( + b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\ + Content-Length: 1048576\r\nConnection: close\r\n\r\n\ + {truncated", + ) + .await; + let _ = sock.shutdown().await; + } + }); + + let http = Client::builder() + .timeout(Duration::from_secs(5)) + .build() + .unwrap(); + let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key") + .await + .unwrap_err(); + assert!( + matches!(&err, AgentError::Llm(s) if s.contains("body read")), + "truncated body must surface as AgentError::Llm with 'body read': got {err:?}" + ); + assert!( + matches!(&err, AgentError::Llm(s) if s.contains("cumulative")), + "body-read error must include terminal_llm_error's cumulative context: got {err:?}" + ); + } + + /// A `TokenSource` whose `refresh_now` always returns the identical token — + /// models a static API key whose bytes never change on refresh. + struct StaticAuth { + token: String, + } + + #[async_trait::async_trait] + impl TokenSource for StaticAuth { + async fn bearer(&self) -> Result { + Ok(self.token.clone()) + } + async fn refresh_now(&self, _rejected: &str) -> Result { + Ok(self.token.clone()) // static: same bytes every time + } + } + + /// A `TokenSource` that returns a stale token from `bearer()` and a + /// distinct fresh token from `refresh_now()`, modelling a PKCE OAuth source. + struct MintingAuth { + stale: String, + fresh: String, + refreshes: std::sync::atomic::AtomicU32, + } + + #[async_trait::async_trait] + impl TokenSource for MintingAuth { + async fn bearer(&self) -> Result { + Ok(self.stale.clone()) + } + async fn refresh_now(&self, _rejected: &str) -> Result { + self.refreshes + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Ok(self.fresh.clone()) + } + } + + /// A static key 401: `refresh_now` returns the same bytes — the second + /// wire request would be byte-identical, so the retry must be skipped. + /// Exactly one wire request reaches the server. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn post_openrouter_static_key_401_single_attempt_no_retry() { + use std::sync::atomic::Ordering; + + let (url, _captured, attempts) = spawn_openrouter_stub(vec![CannedResponse::new( + 401, + r#"{"error":{"message":"invalid api key"}}"#, + )]) + .await; + let auth = Arc::new(StaticAuth { + token: "static-key".into(), + }); + let llm = llm_with(auth); + let mut c = cfg(Provider::OpenRouter); + c.base_url = url; + + let err = llm.post_openrouter(&c, &json!({})).await.unwrap_err(); + assert!( + matches!(&err, AgentError::LlmAuth(s) if s.contains("static key rejected")), + "static 401 must surface as LlmAuth with 'static key rejected': got {err:?}" + ); + assert_eq!( + attempts.load(Ordering::SeqCst), + 1, + "exactly one wire request — no duplicate retry for a static key" + ); + } + + /// A minting-source 401: `refresh_now` produces a distinct fresh token, so + /// the retry is legitimate. The stub accepts the fresh token's second + /// request with 200 and exactly two wire requests reach the server. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn post_openrouter_minting_source_401_retries_with_fresh_token() { + use std::sync::atomic::Ordering; + + // Stub: always 401 for bearer "stale", 200 for anything else. + // We repurpose `spawn_auth_stub` here: it rejects `Bearer stale`, + // accepts `Bearer fresh`. + let always_401 = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let base = spawn_auth_stub(always_401, 401).await; + + let auth = Arc::new(MintingAuth { + stale: "stale".into(), + fresh: "fresh".into(), + refreshes: std::sync::atomic::AtomicU32::new(0), + }); + let llm = llm_with(auth.clone()); + let mut c = cfg(Provider::OpenRouter); + c.base_url = base; + + let result = llm.post_openrouter(&c, &json!({})).await; + // `spawn_auth_stub` returns `{"ok":true}` on success. + assert!( + result.is_ok(), + "minting-source retry should succeed: {result:?}" + ); + assert_eq!( + auth.refreshes.load(Ordering::SeqCst), + 1, + "exactly one refresh" + ); + } } diff --git a/crates/buzz-agent/src/types.rs b/crates/buzz-agent/src/types.rs index 31172def6d..343a75bf72 100644 --- a/crates/buzz-agent/src/types.rs +++ b/crates/buzz-agent/src/types.rs @@ -62,6 +62,7 @@ pub enum HistoryItem { Assistant { text: String, tool_calls: Vec, + reasoning_details: Option, }, ToolResult(ToolResult), } @@ -83,7 +84,11 @@ impl HistoryItem { fn size_with(&self, content_size: fn(&ToolResultContent) -> usize) -> usize { match self { Self::User(s) => s.len(), - Self::Assistant { text, tool_calls } => { + Self::Assistant { + text, + tool_calls, + reasoning_details, + } => { text.len() + tool_calls .iter() @@ -102,6 +107,11 @@ impl HistoryItem { .unwrap_or(0) }) .sum::() + + reasoning_details + .as_ref() + .and_then(|v| serde_json::to_vec(v).ok()) + .map(|b| b.len()) + .unwrap_or(0) } Self::ToolResult(r) => { r.provider_id.len() + r.content.iter().map(content_size).sum::() @@ -188,6 +198,10 @@ pub struct LlmResponse { /// /// Empty string when the provider returned no reasoning content. pub reasoning: String, + /// Raw `reasoning_details` array from an OpenRouter response, if present. + /// Replayed on subsequent turns so the model can continue its chain-of-thought. + /// `None` for all non-OpenRouter providers. + pub reasoning_details: Option, } #[derive(Debug, Clone, Copy, PartialEq)] @@ -481,6 +495,7 @@ mod tests { arguments: Value::Null, provider_extra: extra, }], + reasoning_details: None, }; let without_extra = HistoryItem::Assistant { text: String::new(), @@ -490,6 +505,7 @@ mod tests { arguments: Value::Null, provider_extra: Map::new(), }], + reasoning_details: None, }; assert!(with_extra.estimated_bytes() > without_extra.estimated_bytes() + 500); assert_eq!( diff --git a/desktop/src-tauri/src/commands/agent_models.rs b/desktop/src-tauri/src/commands/agent_models.rs index ca1fe9bdf6..7ce03b140b 100644 --- a/desktop/src-tauri/src/commands/agent_models.rs +++ b/desktop/src-tauri/src/commands/agent_models.rs @@ -99,6 +99,17 @@ pub async fn get_agent_models( // so a build-provided provider still gets live discovery. let effective_provider = effective_discovery_provider(saved_provider.as_deref(), provider_env_var, &merged_env); + if let Some(models) = discover_openrouter_models( + &state.http_client, + &effective_provider, + &merged_env, + persisted_model.clone(), + ) + .await? + { + return Ok(models); + } + if let Some(models) = discover_openai_compatible_models( &state.http_client, &effective_provider, @@ -154,69 +165,11 @@ fn model_discovery_error(pubkey: &str, error: &str) -> String { ) } -/// Everything `get_agent_models` needs from the record + context, resolved in -/// one pure step so the linked-agent regression test can bind the exact values -/// the command consumes. -#[derive(Debug, PartialEq, Eq)] -struct AgentModelDiscoveryConfig { - /// Effective harness command (descriptor-resolved), for `resolve_command`. - command: String, - /// Effective harness args (descriptor-resolved). - args: Vec, - /// Model from the authoritative resolver spawn uses — linked instances - /// read their definition, never stale `record.model` bytes. - model: Option, - /// Provider from the same authoritative resolver — never stale - /// `record.provider` bytes for linked instances. - provider: Option, - /// The runtime's provider env var (e.g. `GOOSE_PROVIDER`), so discovery - /// can recover the provider from the env when the resolver yields none. - /// `None` for runtimes that do not take a provider, or an unknown command. - provider_env_var: Option<&'static str>, - /// The descriptor's fully layered env (definition/persona/global/agent). - env: BTreeMap, -} - -/// Resolve the model-discovery config for a saved agent — the descriptor-backed -/// successor to the old `saved_agent_model_discovery_config`. -/// -/// Command/args/env come from `resolve_effective_harness_descriptor` (the same -/// resolver as `spawn_agent_child`); model/provider come from -/// `resolve_effective_model_provider` (#1968's definition-authoritative -/// contract) — linked instances read their definition, never a stale -/// materialized `record.model`/`record.provider`, so discovery cannot query a -/// provider this agent will not actually launch with. Definition-less -/// instances keep their own record values, matching spawn's -/// `resolve_definition_less` arm. When the resolver yields no provider, -/// `effective_discovery_provider` recovers the provider the agent will -/// actually launch with from the runtime's own provider env var, read out of -/// the descriptor env (which already layers definition/persona/global values -/// the same way spawn does). -/// -/// Returns `Err("DANGLING_HARNESS_ID:")` from the descriptor resolver when -/// the harness id no longer exists; the caller routes it through -/// `model_discovery_error`. -fn agent_model_discovery_config( - record: &crate::managed_agents::ManagedAgentRecord, - personas: &[crate::managed_agents::AgentDefinition], - global: &crate::managed_agents::GlobalAgentConfig, -) -> Result { - let descriptor = - crate::managed_agents::resolve_effective_harness_descriptor(record, personas, global)?; - let (model, provider) = - crate::managed_agents::resolve_effective_model_provider(record, personas, global); - let provider_env_var = - known_acp_runtime(&descriptor.command).and_then(|meta| meta.provider_env_var); - - Ok(AgentModelDiscoveryConfig { - command: descriptor.command, - args: descriptor.args, - model, - provider, - provider_env_var, - env: descriptor.env, - }) -} +#[path = "agent_models_discovery_config.rs"] +mod discovery_config; +use discovery_config::{ + agent_model_discovery_config, draft_agent_model_discovery_env, AgentModelDiscoveryConfig, +}; #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] @@ -269,31 +222,12 @@ pub async fn discover_agent_models( .unwrap_or_else(|| agent_command.to_string()); let runtime_meta = known_acp_runtime(agent_command); - let mut derived_env = BTreeMap::new(); - if let Some(meta) = runtime_meta { - let provider = input - .provider - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty()); - if !meta.provider_locked { - if let (Some(env_key), Some(provider)) = (meta.provider_env_var, provider) { - derived_env.insert(env_key.to_string(), provider.to_string()); - } - } - } - // Layer definition_env below user env_vars so user overrides always win. - // Reserved keys are stripped, matching the same filter applied at spawn. - let mut filtered_definition_env = BTreeMap::new(); - for (key, value) in &input.definition_env { - if !crate::managed_agents::is_reserved_env_key(key) { - filtered_definition_env.insert(key.clone(), value.clone()); - } - } - // Merge: derived (metadata) → definition env → user env_vars. - let merged_with_def = - crate::managed_agents::merged_user_env(&derived_env, &filtered_definition_env); - let merged_env = crate::managed_agents::merged_user_env(&merged_with_def, &input.env_vars); + let merged_env = draft_agent_model_discovery_env( + agent_command, + input.provider.as_deref(), + &input.definition_env, + &input.env_vars, + ); let merged_env = discovery_env_with_baked_floor(merged_env); // Recover a build-provided provider when the form has none, so the create // dialog discovers live models instead of falling through to the subprocess. @@ -348,6 +282,13 @@ pub async fn discover_agent_models( return Err("Buzz shared compute is not available in this build".to_string()); } + if let Some(models) = + discover_openrouter_models(&state.http_client, &effective_provider, &merged_env, None) + .await? + { + return Ok(models); + } + if let Some(models) = discover_openai_compatible_models( &state.http_client, &effective_provider, @@ -388,6 +329,15 @@ struct OpenAiModelListItem { created: Option, } +#[path = "agent_models_openrouter.rs"] +mod openrouter; +use openrouter::discover_openrouter_models; +#[cfg(test)] +use openrouter::{ + filter_openrouter_models, is_openrouter_provider, openrouter_models_url, + OpenRouterModelListItem, OpenRouterModelListResponse, +}; + fn is_openai_compatible_provider(provider: Option<&str>) -> bool { matches!( provider diff --git a/desktop/src-tauri/src/commands/agent_models_discovery_config.rs b/desktop/src-tauri/src/commands/agent_models_discovery_config.rs new file mode 100644 index 0000000000..e43f09495b --- /dev/null +++ b/desktop/src-tauri/src/commands/agent_models_discovery_config.rs @@ -0,0 +1,112 @@ +//! Model-discovery configuration resolution for the agent-models commands. +//! +//! Two entry points, one per call shape: [`agent_model_discovery_config`] +//! resolves a *saved* agent through the same descriptor/model resolvers spawn +//! uses, and [`draft_agent_model_discovery_env`] derives the env for an unsaved +//! form. Both are pure so the regression tests can bind the exact values the +//! commands consume. +//! +//! Included from `agent_models.rs` via `#[path]`, so `super::*` resolves +//! against that module (the `agent_models_tests.rs` convention). + +use std::collections::BTreeMap; + +use crate::managed_agents::known_acp_runtime; + +/// Everything `get_agent_models` needs from the record + context, resolved in +/// one pure step so the linked-agent regression test can bind the exact values +/// the command consumes. +#[derive(Debug, PartialEq, Eq)] +pub(super) struct AgentModelDiscoveryConfig { + /// Effective harness command (descriptor-resolved), for `resolve_command`. + pub(super) command: String, + /// Effective harness args (descriptor-resolved). + pub(super) args: Vec, + /// Model from the authoritative resolver spawn uses — linked instances + /// read their definition, never stale `record.model` bytes. + pub(super) model: Option, + /// Provider from the same authoritative resolver — never stale + /// `record.provider` bytes for linked instances. + pub(super) provider: Option, + /// The runtime's provider env var (e.g. `GOOSE_PROVIDER`), so discovery + /// can recover the provider from the env when the resolver yields none. + /// `None` for runtimes that do not take a provider, or an unknown command. + pub(super) provider_env_var: Option<&'static str>, + /// The descriptor's fully layered env (definition/persona/global/agent). + pub(super) env: BTreeMap, +} + +/// Resolve the model-discovery config for a saved agent — the descriptor-backed +/// successor to the old `saved_agent_model_discovery_config`. +/// +/// Command/args/env come from `resolve_effective_harness_descriptor` (the same +/// resolver as `spawn_agent_child`); model/provider come from +/// `resolve_effective_model_provider` (#1968's definition-authoritative +/// contract) — linked instances read their definition, never a stale +/// materialized `record.model`/`record.provider`, so discovery cannot query a +/// provider this agent will not actually launch with. Definition-less +/// instances keep their own record values, matching spawn's +/// `resolve_definition_less` arm. When the resolver yields no provider, +/// `effective_discovery_provider` recovers the provider the agent will +/// actually launch with from the runtime's own provider env var, read out of +/// the descriptor env (which already layers definition/persona/global values +/// the same way spawn does). +/// +/// Returns `Err("DANGLING_HARNESS_ID:")` from the descriptor resolver when +/// the harness id no longer exists; the caller routes it through +/// `model_discovery_error`. +pub(super) fn agent_model_discovery_config( + record: &crate::managed_agents::ManagedAgentRecord, + personas: &[crate::managed_agents::AgentDefinition], + global: &crate::managed_agents::GlobalAgentConfig, +) -> Result { + let descriptor = + crate::managed_agents::resolve_effective_harness_descriptor(record, personas, global)?; + let (model, provider) = + crate::managed_agents::resolve_effective_model_provider(record, personas, global); + let provider_env_var = + known_acp_runtime(&descriptor.command).and_then(|meta| meta.provider_env_var); + + Ok(AgentModelDiscoveryConfig { + command: descriptor.command, + args: descriptor.args, + model, + provider, + provider_env_var, + env: descriptor.env, + }) +} + +/// Derive the discovery env for an unsaved ("draft") agent configuration. +/// +/// Mirrors the layering `agent_model_discovery_config` takes from the harness +/// descriptor, but sources the provider from form input: runtime-derived +/// provider env var → definition env → user env vars, so user overrides always +/// win. Extracted so the draft path has the same tested seam as the saved one. +pub(super) fn draft_agent_model_discovery_env( + agent_command: &str, + provider: Option<&str>, + definition_env: &BTreeMap, + env_vars: &BTreeMap, +) -> BTreeMap { + let mut derived_env = BTreeMap::new(); + if let Some(meta) = known_acp_runtime(agent_command) { + let provider = provider.map(str::trim).filter(|value| !value.is_empty()); + if !meta.provider_locked { + if let (Some(env_key), Some(provider)) = (meta.provider_env_var, provider) { + derived_env.insert(env_key.to_string(), provider.to_string()); + } + } + } + // Reserved keys are stripped from definition env, matching the same filter + // applied at spawn. + let mut filtered_definition_env = BTreeMap::new(); + for (key, value) in definition_env { + if !crate::managed_agents::is_reserved_env_key(key) { + filtered_definition_env.insert(key.clone(), value.clone()); + } + } + let merged_with_def = + crate::managed_agents::merged_user_env(&derived_env, &filtered_definition_env); + crate::managed_agents::merged_user_env(&merged_with_def, env_vars) +} diff --git a/desktop/src-tauri/src/commands/agent_models_openrouter.rs b/desktop/src-tauri/src/commands/agent_models_openrouter.rs new file mode 100644 index 0000000000..be6dd2cf26 --- /dev/null +++ b/desktop/src-tauri/src/commands/agent_models_openrouter.rs @@ -0,0 +1,112 @@ +use std::collections::BTreeMap; + +use serde::Deserialize; + +use crate::managed_agents::{AgentModelInfo, AgentModelsResponse}; + +#[cfg(test)] +use super::env_value; +use super::{env_or_process_value, redaction_env_with_value, DiscoveryProvider}; + +#[derive(Debug, Deserialize)] +#[cfg_attr(test, derive(Clone))] +pub(super) struct OpenRouterModelListResponse { + pub data: Vec, +} + +#[derive(Debug, Deserialize)] +#[cfg_attr(test, derive(Clone))] +pub(super) struct OpenRouterModelListItem { + pub id: String, + #[serde(default)] + pub supported_parameters: Vec, +} + +pub(super) fn is_openrouter_provider(provider: Option<&str>) -> bool { + matches!( + provider + .map(str::trim) + .map(str::to_ascii_lowercase) + .as_deref(), + Some("openrouter") + ) +} + +#[cfg(test)] +pub(super) fn openrouter_models_url(env: &BTreeMap) -> String { + let base_url = env_value(env, "OPENROUTER_BASE_URL") + .unwrap_or_else(|| "https://openrouter.ai/api/v1".to_string()); + format!("{}/models", base_url.trim_end_matches('/')) +} + +fn openrouter_models_url_for_discovery(env: &BTreeMap) -> String { + let base_url = env_or_process_value(env, "OPENROUTER_BASE_URL") + .unwrap_or_else(|| "https://openrouter.ai/api/v1".to_string()); + format!("{}/models", base_url.trim_end_matches('/')) +} + +pub(super) async fn discover_openrouter_models( + client: &reqwest::Client, + provider: &DiscoveryProvider, + env: &BTreeMap, + selected_model: Option, +) -> Result, String> { + if !is_openrouter_provider(provider.as_deref()) { + return Ok(None); + } + + let api_key = match provider.required_env(env, "OPENROUTER_API_KEY")? { + Some(api_key) => api_key, + None => return Ok(None), + }; + let redaction_env = redaction_env_with_value(env, "OPENROUTER_API_KEY", &api_key); + let url = openrouter_models_url_for_discovery(env); + let response = client + .get(&url) + .bearer_auth(&api_key) + .send() + .await + .map_err(|error| format!("OpenRouter model discovery request failed: {error}"))?; + let status = response.status(); + if !status.is_success() { + let body = response.text().await.unwrap_or_default(); + let body = crate::managed_agents::redact_env_values_in(&body, &redaction_env); + return Err(format!("OpenRouter model discovery HTTP {status}: {body}")); + } + + let response = response + .json::() + .await + .map_err(|error| format!("OpenRouter model discovery response parse failed: {error}"))?; + + filter_openrouter_models(response, selected_model) +} + +pub(super) fn filter_openrouter_models( + response: OpenRouterModelListResponse, + selected_model: Option, +) -> Result, String> { + let models: Vec = response + .data + .into_iter() + .filter(|m| m.supported_parameters.iter().any(|p| p == "tools")) + .map(|m| AgentModelInfo { + id: m.id.clone(), + name: Some(m.id), + description: None, + }) + .collect(); + + if models.is_empty() { + return Err("OpenRouter model discovery returned no tools-capable models".to_string()); + } + + Ok(Some(AgentModelsResponse { + agent_name: "openrouter".to_string(), + agent_version: "models-api".to_string(), + models, + agent_default_model: None, + selected_model, + supports_switching: true, + })) +} diff --git a/desktop/src-tauri/src/commands/agent_models_tests.rs b/desktop/src-tauri/src/commands/agent_models_tests.rs index b65f240900..14c981d730 100644 --- a/desktop/src-tauri/src/commands/agent_models_tests.rs +++ b/desktop/src-tauri/src/commands/agent_models_tests.rs @@ -590,3 +590,294 @@ fn model_discovery_error_converts_dangling_sentinel_to_sentence() { let plain = model_discovery_error("agent-pk", "plain failure"); assert_eq!(plain, "cannot discover models for agent-pk: plain failure"); } + +// --------------------------------------------------------------------------- +// OpenRouter provider +// --------------------------------------------------------------------------- + +#[test] +fn is_openrouter_provider_matches() { + assert!(is_openrouter_provider(Some("openrouter"))); + assert!(is_openrouter_provider(Some(" OpenRouter "))); + assert!(!is_openrouter_provider(Some("openai"))); + assert!(!is_openrouter_provider(Some("anthropic"))); + assert!(!is_openrouter_provider(None)); +} + +#[test] +fn openrouter_models_url_uses_default_base_url() { + assert_eq!( + openrouter_models_url(&BTreeMap::new()), + "https://openrouter.ai/api/v1/models" + ); +} + +#[test] +fn openrouter_models_url_respects_custom_base_url() { + let env = BTreeMap::from([( + "OPENROUTER_BASE_URL".to_string(), + "https://eu.openrouter.ai/api/v1".to_string(), + )]); + assert_eq!( + openrouter_models_url(&env), + "https://eu.openrouter.ai/api/v1/models" + ); +} + +#[test] +fn openrouter_models_url_strips_trailing_slash() { + let env = BTreeMap::from([( + "OPENROUTER_BASE_URL".to_string(), + "https://proxy.example.com/api/v1/".to_string(), + )]); + assert_eq!( + openrouter_models_url(&env), + "https://proxy.example.com/api/v1/models" + ); +} + +#[test] +fn openrouter_filter_keeps_tools_capable_models() { + let response = OpenRouterModelListResponse { + data: vec![ + OpenRouterModelListItem { + id: "anthropic/claude-opus-4-7".to_string(), + supported_parameters: vec!["tools".to_string(), "reasoning".to_string()], + }, + OpenRouterModelListItem { + id: "openai/gpt-5.5-pro".to_string(), + supported_parameters: vec!["tools".to_string()], + }, + OpenRouterModelListItem { + id: "meta-llama/llama-no-tools".to_string(), + supported_parameters: vec!["temperature".to_string()], + }, + ], + }; + let result = filter_openrouter_models(response, None).unwrap().unwrap(); + let ids: Vec<_> = result.models.iter().map(|m| m.id.as_str()).collect(); + assert_eq!(ids, vec!["anthropic/claude-opus-4-7", "openai/gpt-5.5-pro"]); +} + +#[test] +fn openrouter_filter_excludes_absent_supported_parameters() { + let response: OpenRouterModelListResponse = + serde_json::from_str(r#"{"data": [{"id": "model-no-params"}]}"#).unwrap(); + assert!( + response.data[0].supported_parameters.is_empty(), + "absent supported_parameters must default to empty vec" + ); + let result = filter_openrouter_models(response, None); + assert!( + result.is_err(), + "models with no supported_parameters must be excluded" + ); + assert!( + result.unwrap_err().contains("no tools-capable models"), + "error must indicate no tools-capable models" + ); +} + +#[test] +fn openrouter_filter_excludes_empty_supported_parameters() { + let response = OpenRouterModelListResponse { + data: vec![OpenRouterModelListItem { + id: "model-empty-params".to_string(), + supported_parameters: Vec::new(), + }], + }; + let result = filter_openrouter_models(response, None); + assert!(result.is_err()); + assert!(result.unwrap_err().contains("no tools-capable models")); +} + +#[test] +fn openrouter_filter_empty_result_returns_error() { + let response = OpenRouterModelListResponse { data: Vec::new() }; + let result = filter_openrouter_models(response, None); + assert!(result.is_err()); + assert!(result.unwrap_err().contains("no tools-capable models")); +} + +#[test] +fn openrouter_filter_preserves_selected_model() { + let response = OpenRouterModelListResponse { + data: vec![OpenRouterModelListItem { + id: "openai/gpt-5.5-pro".to_string(), + supported_parameters: vec!["tools".to_string()], + }], + }; + let result = filter_openrouter_models(response, Some("openai/gpt-5.5-pro".to_string())) + .unwrap() + .unwrap(); + assert_eq!(result.selected_model.as_deref(), Some("openai/gpt-5.5-pro")); +} + +#[test] +fn openrouter_credential_redaction_env_records_key() { + let env = BTreeMap::from([( + "OPENROUTER_API_KEY".to_string(), + "sk-or-v1-secret-key-12345".to_string(), + )]); + let redaction = + redaction_env_with_value(&env, "OPENROUTER_API_KEY", "sk-or-v1-secret-key-12345"); + assert_eq!( + redaction.get("OPENROUTER_API_KEY").map(String::as_str), + Some("sk-or-v1-secret-key-12345"), + "redaction env must record the API key for error body redaction" + ); +} + +#[test] +fn openrouter_saved_agent_model_discovery_resolves_provider() { + let record: crate::managed_agents::ManagedAgentRecord = serde_json::from_str( + r#"{ + "pubkey": "abcd1234", + "name": "test-agent", + "private_key_nsec": "nsec1fake", + "relay_url": "wss://localhost:3000", + "acp_command": "buzz-acp", + "agent_command": "buzz-agent", + "agent_command_override": "buzz-agent", + "agent_args": [], + "mcp_command": "", + "turn_timeout_seconds": 320, + "system_prompt": null, + "model": "anthropic/claude-sonnet-4", + "provider": "openrouter", + "env_vars": { + "OPENROUTER_API_KEY": "sk-or-test-key", + "BUZZ_PRIVATE_KEY": "must-not-leak" + }, + "created_at": "2026-01-01T00:00:00Z", + "updated_at": "2026-01-01T00:00:00Z", + "last_started_at": null, + "last_stopped_at": null, + "last_exit_code": null, + "last_error": null + }"#, + ) + .expect("sample openrouter managed agent record"); + + let discovery = agent_model_discovery_config( + &record, + &[], + &crate::managed_agents::GlobalAgentConfig::default(), + ) + .expect("discovery config should resolve for an openrouter record"); + assert_eq!(discovery.provider.as_deref(), Some("openrouter")); + assert_eq!( + discovery.model.as_deref(), + Some("anthropic/claude-sonnet-4") + ); + assert_eq!( + discovery.env.get("OPENROUTER_API_KEY").map(String::as_str), + Some("sk-or-test-key") + ); + assert!(!discovery.env.contains_key("BUZZ_PRIVATE_KEY")); +} + +/// B5/T4: unsaved-agent ("draft") discovery mirrors the saved-agent path — +/// `draft_agent_model_discovery_env` must derive the provider env var from +/// form input the same way `agent_model_discovery_config` derives it from a +/// persisted record's harness descriptor, and preserve caller-supplied env +/// (including the OpenRouter API key) unmodified. +#[test] +fn openrouter_draft_agent_model_discovery_derives_provider_env() { + let env_vars = BTreeMap::from([( + "OPENROUTER_API_KEY".to_string(), + "sk-or-draft-key".to_string(), + )]); + + let merged = draft_agent_model_discovery_env( + "buzz-agent", + Some("openrouter"), + &BTreeMap::new(), + &env_vars, + ); + + assert_eq!( + merged.get("BUZZ_AGENT_PROVIDER").map(String::as_str), + Some("openrouter"), + "provider env var must be derived from form input for a known ACP runtime" + ); + assert_eq!( + merged.get("OPENROUTER_API_KEY").map(String::as_str), + Some("sk-or-draft-key"), + "caller-supplied env vars must survive the merge" + ); +} + +#[test] +fn draft_agent_model_discovery_env_omits_provider_when_absent() { + let merged = + draft_agent_model_discovery_env("buzz-agent", None, &BTreeMap::new(), &BTreeMap::new()); + assert!( + !merged.contains_key("BUZZ_AGENT_PROVIDER"), + "no provider must be derived when the caller supplies none" + ); +} + +/// The three-tier precedence this merge exists to preserve: main's inline +/// `derived → definition_env → env_vars` layering was folded into +/// `draft_agent_model_discovery_env`, so pin the order at every collision +/// boundary rather than trusting the two single-tier tests above. +/// +/// `SHARED` collides across all three tiers, so the user value proves the +/// full chain; the pairwise keys prove each adjacent boundary independently +/// (a merge that dropped only the middle tier would still satisfy `SHARED`). +/// `BUZZ_PRIVATE_KEY` proves a reserved key cannot ride in on a harness +/// definition, which is the tier a user never types. +#[test] +fn draft_agent_model_discovery_env_layers_all_three_tiers_in_order() { + // Tier 2 (middle): harness definition env — overlays the runtime-derived + // floor, loses to user env. + let definition_env = BTreeMap::from([ + ("SHARED".to_string(), "from-definition".to_string()), + // Collides with tier 1: `buzz-agent`'s own provider env var, which the + // `provider` argument derives below. + ("BUZZ_AGENT_PROVIDER".to_string(), "openai".to_string()), + ("USER_OVER_DEF".to_string(), "from-definition".to_string()), + ("DEFINITION_ONLY".to_string(), "from-definition".to_string()), + // Reserved: must never reach the child, even from a definition. + ("BUZZ_PRIVATE_KEY".to_string(), "must-not-leak".to_string()), + ]); + // Tier 3 (top): user-entered env — wins over everything. + let env_vars = BTreeMap::from([ + ("SHARED".to_string(), "from-user".to_string()), + ("USER_OVER_DEF".to_string(), "from-user".to_string()), + ("USER_ONLY".to_string(), "from-user".to_string()), + ]); + + // Tier 1 (floor): `Some("openrouter")` derives BUZZ_AGENT_PROVIDER. + let merged = draft_agent_model_discovery_env( + "buzz-agent", + Some("openrouter"), + &definition_env, + &env_vars, + ); + + let expected: &[(&str, Option<&str>)] = &[ + // Collides in all three tiers — the top tier wins. + ("SHARED", Some("from-user")), + // Tier 2 over tier 1: the definition's value survives, proving the + // derived provider is the floor and not layered on top. + ("BUZZ_AGENT_PROVIDER", Some("openai")), + // Tier 3 over tier 2. + ("USER_OVER_DEF", Some("from-user")), + // Single-tier keys pass through untouched. + ("DEFINITION_ONLY", Some("from-definition")), + ("USER_ONLY", Some("from-user")), + // Reserved keys never survive the definition tier. Doubly enforced — + // the explicit `is_reserved_env_key` filter here and `merged_user_env`'s + // own `retain` — so this pins the contract, not either mechanism. + ("BUZZ_PRIVATE_KEY", None), + ]; + for (key, want) in expected { + assert_eq!( + merged.get(*key).map(String::as_str), + *want, + "env key `{key}` must resolve to {want:?} after three-tier layering" + ); + } +} diff --git a/desktop/src-tauri/src/managed_agents/readiness.rs b/desktop/src-tauri/src/managed_agents/readiness.rs index c053d933c5..fa8eb36fa1 100644 --- a/desktop/src-tauri/src/managed_agents/readiness.rs +++ b/desktop/src-tauri/src/managed_agents/readiness.rs @@ -481,6 +481,7 @@ fn buzz_agent_requirements(effective: &EffectiveAgentEnv) -> Vec { } Some("anthropic") => Some("ANTHROPIC_MODEL"), Some("openai") | Some("openai-compat") => Some("OPENAI_COMPAT_MODEL"), + Some("openrouter") => Some("OPENROUTER_MODEL"), _ => None, }; let model_present = effective @@ -523,6 +524,12 @@ fn buzz_agent_requirements(effective: &EffectiveAgentEnv) -> Vec { key: "DATABRICKS_HOST".to_string(), }); } + Some("openrouter") + if env_key_missing("OPENROUTER_API_KEY") => { + missing.push(Requirement::EnvKey { + key: "OPENROUTER_API_KEY".to_string(), + }); + } _ => { // Unknown provider or no provider yet — only the NormalizedField // requirement above captures this gap. @@ -630,6 +637,13 @@ fn goose_requirements( key: "DATABRICKS_HOST".to_string(), }); } + Some("openrouter") + if env_key_missing("OPENROUTER_API_KEY") && !file_key_present("OPENROUTER_API_KEY") => + { + missing.push(Requirement::EnvKey { + key: "OPENROUTER_API_KEY".to_string(), + }); + } _ => {} } @@ -1668,195 +1682,62 @@ mod tests { field: "model".to_string() })); } -} - -// ── goose file-config–aware requirement tests ───────────────────────────── -// -// These tests call `goose_requirements` directly, injecting a synthetic -// `RuntimeFileConfig` so there is no disk I/O and tests are deterministic. - -#[cfg(test)] -mod goose_file_config_tests { - use std::collections::BTreeMap; - - use super::*; - use crate::managed_agents::config_bridge::RuntimeFileConfig; - - fn empty_env() -> EffectiveAgentEnv { - EffectiveAgentEnv { - env: BTreeMap::new(), - config_file_path: Some("~/.config/goose/config.yaml"), - effective_command: "goose".to_string(), - } - } - fn env_with(pairs: &[(&str, &str)]) -> EffectiveAgentEnv { - EffectiveAgentEnv { - env: pairs - .iter() - .map(|(k, v)| (k.to_string(), v.to_string())) - .collect(), - config_file_path: Some("~/.config/goose/config.yaml"), - effective_command: "goose".to_string(), - } - } - - fn databricks_file_config() -> RuntimeFileConfig { - let mut extra = BTreeMap::new(); - extra.insert( - "DATABRICKS_HOST".to_string(), - "https://dbc.example.com".to_string(), - ); - RuntimeFileConfig { - provider: Some("databricks_v2".to_string()), - model: Some("goose-claude-4-6-opus".to_string()), - extra, - ..Default::default() - } - } + // ── OpenRouter readiness ───────────────────────────────────────────── #[test] - fn goose_file_config_silences_databricks_host_requirement() { - // File has provider, model, and DATABRICKS_HOST — all requirements silenced. - let env = empty_env(); - let cfg = databricks_file_config(); - let result = goose_requirements(&env, Some(&cfg)); - assert!( - result.is_empty(), - "all requirements should be silenced by goose file config; \ - got: {:?}", - result - ); - } - - #[test] - fn goose_env_empty_file_absent_still_not_ready() { - // No env, no file config → provider and model both required. - let env = empty_env(); - let result = goose_requirements(&env, None); - assert!( - result.contains(&Requirement::NormalizedField { - field: "provider".to_string() - }), - "provider must be required when absent from both env and file" - ); - assert!( - result.contains(&Requirement::NormalizedField { - field: "model".to_string() - }), - "model must be required when absent from both env and file" - ); - } - - #[test] - fn goose_file_config_silences_provider_and_model_but_not_anthropic_key() { - // File has provider=anthropic and model, but ANTHROPIC_API_KEY is not - // in the file's `extra` map — it must still be required. - let cfg = RuntimeFileConfig { - provider: Some("anthropic".to_string()), - model: Some("claude-opus-4-5".to_string()), - extra: BTreeMap::new(), - ..Default::default() - }; - let env = empty_env(); - let result = goose_requirements(&env, Some(&cfg)); - // Provider and model silenced. - assert!( - !result.contains(&Requirement::NormalizedField { - field: "provider".to_string() - }), - "provider silenced by file config" - ); - assert!( - !result.contains(&Requirement::NormalizedField { - field: "model".to_string() - }), - "model silenced by file config" + fn buzz_agent_openrouter_with_all_fields_is_ready() { + let env = make_env( + "buzz-agent", + env_with(&[ + ("BUZZ_AGENT_PROVIDER", "openrouter"), + ("BUZZ_AGENT_MODEL", "anthropic/claude-sonnet-4"), + ("OPENROUTER_API_KEY", "sk-or-test-key"), + ]), ); - // ANTHROPIC_API_KEY not in file extra → still required. + let result = agent_readiness(&env); assert!( - result.contains(&Requirement::EnvKey { - key: "ANTHROPIC_API_KEY".to_string() - }), - "ANTHROPIC_API_KEY must remain required when not in file extra" + result.is_ready(), + "openrouter with all fields should be ready" ); } #[test] - fn goose_env_provider_wins_over_file_provider_for_cred_check() { - // Env has GOOSE_PROVIDER=anthropic (different from file's databricks_v2). - // The env provider must win for credential checking. - let env = env_with(&[ - ("GOOSE_PROVIDER", "anthropic"), - ("GOOSE_MODEL", "claude-opus-4-5"), - ]); - let cfg = databricks_file_config(); // has provider=databricks_v2 - let result = goose_requirements(&env, Some(&cfg)); - // anthropic requires ANTHROPIC_API_KEY, not DATABRICKS_HOST. - assert!( - result.contains(&Requirement::EnvKey { - key: "ANTHROPIC_API_KEY".to_string() - }), - "env provider=anthropic must require ANTHROPIC_API_KEY" - ); - assert!( - !result.contains(&Requirement::EnvKey { - key: "DATABRICKS_HOST".to_string() - }), - "env provider=anthropic must NOT require DATABRICKS_HOST" + fn buzz_agent_openrouter_missing_key_returns_not_ready() { + let env = make_env( + "buzz-agent", + env_with(&[ + ("BUZZ_AGENT_PROVIDER", "openrouter"), + ("BUZZ_AGENT_MODEL", "anthropic/claude-sonnet-4"), + ]), ); + let result = agent_readiness(&env); + assert!(!result.is_ready()); + assert!(result.requirements().contains(&Requirement::EnvKey { + key: "OPENROUTER_API_KEY".to_string() + })); } #[test] - fn goose_flat_databricks_host_in_file_config_silences_requirement() { - // Will's typical goose config: flat DATABRICKS_HOST at the top level, - // no active_provider — provider inferred as "databricks". - // The parser must store extra["DATABRICKS_HOST"] = value (canonical key), - // and goose_requirements must then silence the DATABRICKS_HOST requirement. - let mut extra = BTreeMap::new(); - extra.insert( - "DATABRICKS_HOST".to_string(), - "https://block.cloud.databricks.com".to_string(), - ); - let cfg = RuntimeFileConfig { - provider: Some("databricks".to_string()), - model: Some("goose-claude-4-5".to_string()), - extra, - ..Default::default() - }; - let env = empty_env(); - let result = goose_requirements(&env, Some(&cfg)); - // All requirements silenced — provider (file), model (file), DATABRICKS_HOST (file). - assert!( - result.is_empty(), - "flat DATABRICKS_HOST in file config must silence all requirements; \ - got: {:?}", - result + fn buzz_agent_openrouter_with_provider_model_fallback_is_ready() { + let env = make_env( + "buzz-agent", + env_with(&[ + ("BUZZ_AGENT_PROVIDER", "openrouter"), + ("OPENROUTER_MODEL", "google/gemini-2.5-flash"), + ("OPENROUTER_API_KEY", "sk-or-test-key"), + ]), ); - } - - #[test] - fn goose_goose_provider_databricks_flat_host_silences_databricks_host() { - // GOOSE_PROVIDER=databricks (not active_provider) + flat DATABRICKS_HOST. - // The parser canonicalizes to extra["DATABRICKS_HOST"]; readiness must silence it. - let mut extra = BTreeMap::new(); - extra.insert( - "DATABRICKS_HOST".to_string(), - "https://dbc.example.com".to_string(), - ); - let cfg = RuntimeFileConfig { - provider: Some("databricks".to_string()), - model: Some("some-model".to_string()), - extra, - ..Default::default() - }; - let env = empty_env(); - let result = goose_requirements(&env, Some(&cfg)); + let result = agent_readiness(&env); assert!( - !result.contains(&Requirement::EnvKey { - key: "DATABRICKS_HOST".to_string() - }), - "DATABRICKS_HOST must be silenced when canonical key is in file extra" + result.is_ready(), + "OPENROUTER_MODEL fallback should satisfy model requirement" ); } } + +// Goose file-config-aware requirement tests live in a sibling file so this +// module stays under the desktop file-size ratchet. +#[cfg(test)] +#[path = "readiness_goose_file_config_tests.rs"] +mod goose_file_config_tests; diff --git a/desktop/src-tauri/src/managed_agents/readiness_goose_file_config_tests.rs b/desktop/src-tauri/src/managed_agents/readiness_goose_file_config_tests.rs new file mode 100644 index 0000000000..46d0e4c7a7 --- /dev/null +++ b/desktop/src-tauri/src/managed_agents/readiness_goose_file_config_tests.rs @@ -0,0 +1,190 @@ +//! Goose file-config-aware requirement tests. +//! +//! These tests call `goose_requirements` directly, injecting a synthetic +//! `RuntimeFileConfig` so there is no disk I/O and tests are deterministic. +//! +//! Included from `readiness.rs` via `#[path]`; `super::*` therefore resolves +//! against that module, matching the `storage_tests.rs` convention. + +use std::collections::BTreeMap; + +use super::*; +use crate::managed_agents::config_bridge::RuntimeFileConfig; + +fn empty_env() -> EffectiveAgentEnv { + EffectiveAgentEnv { + env: BTreeMap::new(), + config_file_path: Some("~/.config/goose/config.yaml"), + effective_command: "goose".to_string(), + } +} + +fn env_with(pairs: &[(&str, &str)]) -> EffectiveAgentEnv { + EffectiveAgentEnv { + env: pairs + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(), + config_file_path: Some("~/.config/goose/config.yaml"), + effective_command: "goose".to_string(), + } +} + +fn databricks_file_config() -> RuntimeFileConfig { + let mut extra = BTreeMap::new(); + extra.insert( + "DATABRICKS_HOST".to_string(), + "https://dbc.example.com".to_string(), + ); + RuntimeFileConfig { + provider: Some("databricks_v2".to_string()), + model: Some("goose-claude-4-6-opus".to_string()), + extra, + ..Default::default() + } +} + +#[test] +fn goose_file_config_silences_databricks_host_requirement() { + // File has provider, model, and DATABRICKS_HOST — all requirements silenced. + let env = empty_env(); + let cfg = databricks_file_config(); + let result = goose_requirements(&env, Some(&cfg)); + assert!( + result.is_empty(), + "all requirements should be silenced by goose file config; \ + got: {:?}", + result + ); +} + +#[test] +fn goose_env_empty_file_absent_still_not_ready() { + // No env, no file config → provider and model both required. + let env = empty_env(); + let result = goose_requirements(&env, None); + assert!( + result.contains(&Requirement::NormalizedField { + field: "provider".to_string() + }), + "provider must be required when absent from both env and file" + ); + assert!( + result.contains(&Requirement::NormalizedField { + field: "model".to_string() + }), + "model must be required when absent from both env and file" + ); +} + +#[test] +fn goose_file_config_silences_provider_and_model_but_not_anthropic_key() { + // File has provider=anthropic and model, but ANTHROPIC_API_KEY is not + // in the file's `extra` map — it must still be required. + let cfg = RuntimeFileConfig { + provider: Some("anthropic".to_string()), + model: Some("claude-opus-4-5".to_string()), + extra: BTreeMap::new(), + ..Default::default() + }; + let env = empty_env(); + let result = goose_requirements(&env, Some(&cfg)); + // Provider and model silenced. + assert!( + !result.contains(&Requirement::NormalizedField { + field: "provider".to_string() + }), + "provider silenced by file config" + ); + assert!( + !result.contains(&Requirement::NormalizedField { + field: "model".to_string() + }), + "model silenced by file config" + ); + // ANTHROPIC_API_KEY not in file extra → still required. + assert!( + result.contains(&Requirement::EnvKey { + key: "ANTHROPIC_API_KEY".to_string() + }), + "ANTHROPIC_API_KEY must remain required when not in file extra" + ); +} + +#[test] +fn goose_env_provider_wins_over_file_provider_for_cred_check() { + // Env has GOOSE_PROVIDER=anthropic (different from file's databricks_v2). + // The env provider must win for credential checking. + let env = env_with(&[ + ("GOOSE_PROVIDER", "anthropic"), + ("GOOSE_MODEL", "claude-opus-4-5"), + ]); + let cfg = databricks_file_config(); // has provider=databricks_v2 + let result = goose_requirements(&env, Some(&cfg)); + // anthropic requires ANTHROPIC_API_KEY, not DATABRICKS_HOST. + assert!( + result.contains(&Requirement::EnvKey { + key: "ANTHROPIC_API_KEY".to_string() + }), + "env provider=anthropic must require ANTHROPIC_API_KEY" + ); + assert!( + !result.contains(&Requirement::EnvKey { + key: "DATABRICKS_HOST".to_string() + }), + "env provider=anthropic must NOT require DATABRICKS_HOST" + ); +} + +#[test] +fn goose_flat_databricks_host_in_file_config_silences_requirement() { + // Will's typical goose config: flat DATABRICKS_HOST at the top level, + // no active_provider — provider inferred as "databricks". + // The parser must store extra["DATABRICKS_HOST"] = value (canonical key), + // and goose_requirements must then silence the DATABRICKS_HOST requirement. + let mut extra = BTreeMap::new(); + extra.insert( + "DATABRICKS_HOST".to_string(), + "https://block.cloud.databricks.com".to_string(), + ); + let cfg = RuntimeFileConfig { + provider: Some("databricks".to_string()), + model: Some("goose-claude-4-5".to_string()), + extra, + ..Default::default() + }; + let env = empty_env(); + let result = goose_requirements(&env, Some(&cfg)); + // All requirements silenced — provider (file), model (file), DATABRICKS_HOST (file). + assert!( + result.is_empty(), + "flat DATABRICKS_HOST in file config must silence all requirements; \ + got: {:?}", + result + ); +} + +#[test] +fn goose_goose_provider_databricks_flat_host_silences_databricks_host() { + // GOOSE_PROVIDER=databricks (not active_provider) + flat DATABRICKS_HOST. + // The parser canonicalizes to extra["DATABRICKS_HOST"]; readiness must silence it. + let mut extra = BTreeMap::new(); + extra.insert( + "DATABRICKS_HOST".to_string(), + "https://dbc.example.com".to_string(), + ); + let cfg = RuntimeFileConfig { + provider: Some("databricks".to_string()), + model: Some("some-model".to_string()), + extra, + ..Default::default() + }; + let env = empty_env(); + let result = goose_requirements(&env, Some(&cfg)); + assert!( + !result.contains(&Requirement::EnvKey { + key: "DATABRICKS_HOST".to_string() + }), + "DATABRICKS_HOST must be silenced when canonical key is in file extra" + ); +} diff --git a/desktop/src/features/agents/ui/agentConfigOptions.tsx b/desktop/src/features/agents/ui/agentConfigOptions.tsx index 1313d2cec4..d51c970f29 100644 --- a/desktop/src/features/agents/ui/agentConfigOptions.tsx +++ b/desktop/src/features/agents/ui/agentConfigOptions.tsx @@ -42,6 +42,7 @@ const KNOWN_LLM_PROVIDER_IDS = [ "databricks_v2", "openai", "openai-compat", + "openrouter", ] as const; type PersonaLlmProviderId = (typeof KNOWN_LLM_PROVIDER_IDS)[number]; @@ -109,6 +110,10 @@ const PROVIDER_CREDENTIAL_CONFIG: Partial< "databricks-v2": { requiredEnvKeys: ["DATABRICKS_HOST"], }, + openrouter: { + requiredEnvKeys: ["OPENROUTER_API_KEY"], + secretEnvVar: "OPENROUTER_API_KEY", + }, }; const DEFAULT_MODEL_OPTION: PersonaModelOption = { @@ -120,6 +125,7 @@ export const PERSONA_LLM_PROVIDER_OPTIONS: readonly PersonaModelOption[] = [ { id: "anthropic", label: "Anthropic" }, { id: "openai", label: "OpenAI" }, { id: "openai-compat", label: "OpenAI-compatible" }, + { id: "openrouter", label: "OpenRouter" }, { id: "relay-mesh", label: "Buzz shared compute" }, { id: "databricks", label: "Databricks" }, { id: "databricks_v2", label: "Databricks v2" }, @@ -279,7 +285,8 @@ export function providerRequiresExplicitModel( return ( trimmedProvider === "anthropic" || trimmedProvider === "openai" || - trimmedProvider === "openai-compat" + trimmedProvider === "openai-compat" || + trimmedProvider === "openrouter" ); } diff --git a/desktop/src/features/agents/ui/buzzAgentConfig.ts b/desktop/src/features/agents/ui/buzzAgentConfig.ts index a0271fec0f..be663c35cb 100644 --- a/desktop/src/features/agents/ui/buzzAgentConfig.ts +++ b/desktop/src/features/agents/ui/buzzAgentConfig.ts @@ -128,6 +128,9 @@ export function getProviderEffortConfig( // databricks v1 uses OpenAI Chat Completions wire format. return openaiConfig(m); } + if (provider === "openrouter") { + return { validValues: ALL_VALUES, defaultValue: "medium" }; + } // openai-compat, unknown, empty — all values, default medium. return { validValues: ALL_VALUES, defaultValue: "medium" }; } diff --git a/desktop/src/features/agents/ui/effortTable.fixture.json b/desktop/src/features/agents/ui/effortTable.fixture.json index defb1f86de..d097bc995f 100644 --- a/desktop/src/features/agents/ui/effortTable.fixture.json +++ b/desktop/src/features/agents/ui/effortTable.fixture.json @@ -209,6 +209,13 @@ "validValues": ["none", "minimal", "low", "medium", "high", "xhigh", "max"], "defaultValue": "medium" }, + { + "note": "openrouter: all-7 with medium default", + "provider": "openrouter", + "model": "", + "validValues": ["none", "minimal", "low", "medium", "high", "xhigh", "max"], + "defaultValue": "medium" + }, { "note": "empty provider: all-7 with medium default", "provider": "", From 7012d86d52fd188b27c7beedeaa132d9c1f61fa8 Mon Sep 17 00:00:00 2001 From: Kalvin C Date: Wed, 29 Jul 2026 17:00:41 -0700 Subject: [PATCH 28/32] feat: configure S3 URL addressing style (#3400) ## Summary - add one strict `BUZZ_S3_ADDRESSING_STYLE=path|virtual` setting shared by media and Git/CAS storage - preserve path-style defaults for bundled Compose/Helm MinIO while supporting Railway's virtual-hosted bucket contract - fail startup on invalid or non-Unicode values before dependency connection, and validate the Helm value with the same two choices - document operator mappings and why endpoint and bucket remain separate for routing and SigV4 signing ## Best-practice rationale AWS documents both URL forms and favors virtual-hosted addressing for S3, while compatibility endpoints such as the bundled MinIO deployment can require path style. `rust-s3` defaults to virtual/subdomain addressing and provides `with_path_style()` for the explicit compatibility case. Some providers buckets only support as virtual-hosted bucket styles. This PR therefore uses one explicit, provider-neutral switch rather than endpoint heuristics or fallback behavior, while retaining `path` as Buzz's backward-compatible default. Sources: - https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html - https://docs.rs/rust-s3/0.37.0/s3/bucket/struct.Bucket.html - https://docs.railway.com/storage-buckets#url-style - https://github.com/minio/minio/blob/master/docs/config/README.md#domain ## Validation - `cargo fmt --all` - `cargo check --workspace --all-targets` - targeted `buzz-media` and `buzz-relay` parsing/client-construction tests for defaults, strict errors, and both URL styles - Helm unittest: 45/45 passed - Compose config/render validation passed - local MinIO path-mode relay startup passed the Git A3 conformance probe and became ready - unreachable object storage failed startup and readiness never opened - push hooks completed the broader Rust and desktop suites successfully --------- Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz> Signed-off-by: Kalvin Chau Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz> --- .env.example | 13 +++ Cargo.lock | 1 + crates/buzz-media/src/config.rs | 69 +++++++++++++- crates/buzz-media/src/lib.rs | 2 +- crates/buzz-media/src/storage.rs | 27 +++++- crates/buzz-media/src/upload.rs | 1 + crates/buzz-media/src/validation.rs | 1 + crates/buzz-media/tests/static_creds_minio.rs | 13 ++- crates/buzz-relay/src/api/git/cas_publish.rs | 25 ++---- crates/buzz-relay/src/api/git/hydrate.rs | 14 ++- crates/buzz-relay/src/api/git/store.rs | 70 ++++++++++++--- crates/buzz-relay/src/config.rs | 71 +++++++++++++++ crates/buzz-relay/src/state.rs | 1 + crates/buzz-test-client/Cargo.toml | 1 + crates/buzz-test-client/tests/e2e_git.rs | 90 +++++++++++++++---- deploy/charts/buzz/README.md | 42 +++++++++ deploy/charts/buzz/examples/argocd-app.yaml | 2 + .../buzz/examples/flux-helmrelease.yaml | 2 + deploy/charts/buzz/templates/_validate.tpl | 8 +- deploy/charts/buzz/templates/deployment.yaml | 4 + deploy/charts/buzz/tests/render_test.yaml | 38 ++++++++ deploy/charts/buzz/tests/validation_test.yaml | 11 +++ deploy/charts/buzz/values.schema.json | 9 ++ deploy/charts/buzz/values.yaml | 6 ++ deploy/compose/.env.example | 2 + deploy/compose/README.md | 5 ++ deploy/compose/compose.yml | 2 + .../src/commands/media_snapshot_png.rs | 1 + 28 files changed, 471 insertions(+), 60 deletions(-) diff --git a/.env.example b/.env.example index 3dc54856e7..b9bfcada0e 100644 --- a/.env.example +++ b/.env.example @@ -82,6 +82,19 @@ RELAY_URL=ws://localhost:3000 # BUZZ_GIT_PACK_CACHE_MAX_BYTES=5368709120 # BUZZ_GIT_PACK_CACHE_MAX_CONCURRENT_POPULATIONS=2 +# ----------------------------------------------------------------------------- +# S3-Compatible Object Storage (media + Git/CAS) +# ----------------------------------------------------------------------------- +# The local MinIO container is reachable from host processes at localhost:9000. +# Path style keeps the bucket in the URL path and is required by this local DNS +# setup. Use `virtual` only when the provider requires bucket-as-subdomain URLs. +BUZZ_S3_ENDPOINT=http://localhost:9000 +BUZZ_S3_ACCESS_KEY=buzz_dev +BUZZ_S3_SECRET_KEY=buzz_dev_secret +BUZZ_S3_BUCKET=buzz-media +BUZZ_S3_REGION=us-east-1 +BUZZ_S3_ADDRESSING_STYLE=path + # ----------------------------------------------------------------------------- # Media Upload Admission # ----------------------------------------------------------------------------- diff --git a/Cargo.lock b/Cargo.lock index 3b60dc4579..c3ea86d6b3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1239,6 +1239,7 @@ dependencies = [ "anyhow", "base64", "buzz-core", + "buzz-media", "buzz-sdk", "buzz-ws-client", "chrono", diff --git a/crates/buzz-media/src/config.rs b/crates/buzz-media/src/config.rs index 047c08475e..3c70e4afe1 100644 --- a/crates/buzz-media/src/config.rs +++ b/crates/buzz-media/src/config.rs @@ -1,5 +1,38 @@ //! Media storage configuration. +use std::str::FromStr; + +/// S3 URL addressing style shared by media and Git/CAS storage. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum S3AddressingStyle { + /// Put the bucket in the request path (`https://endpoint/bucket/key`). + /// + /// This preserves compatibility with the bundled MinIO deployments, whose + /// internal DNS only resolves the endpoint hostname. + #[default] + Path, + /// Put the bucket in the hostname (`https://bucket.endpoint/key`). + /// + /// This is the standard S3 form and is required by providers such as new + /// Railway Storage Buckets. + Virtual, +} + +impl FromStr for S3AddressingStyle { + type Err = String; + + fn from_str(value: &str) -> Result { + match value { + "path" => Ok(Self::Path), + "virtual" => Ok(Self::Virtual), + _ => Err(format!( + "BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual', got {value:?}" + )), + } + } +} + fn default_max_video_bytes() -> u64 { 524_288_000 // 500 MB } @@ -31,6 +64,9 @@ pub struct MediaConfig { /// the value is not meaningfully checked. #[serde(default = "default_s3_region")] pub s3_region: String, + /// S3 URL addressing style. Defaults to path style for MinIO compatibility. + #[serde(default)] + pub s3_addressing_style: S3AddressingStyle, /// Maximum upload size for images (bytes). Default: 50 MB. pub max_image_bytes: u64, /// Maximum upload size for animated GIFs (bytes). Default: 10 MB. @@ -123,7 +159,8 @@ impl MediaConfig { #[cfg(test)] mod tests { - use super::MediaConfig; + use super::{MediaConfig, S3AddressingStyle}; + use std::str::FromStr; fn valid_config() -> MediaConfig { MediaConfig { @@ -132,6 +169,7 @@ mod tests { s3_secret_key: "s".to_string(), s3_bucket: "buzz-media".to_string(), s3_region: "us-east-1".to_string(), + s3_addressing_style: S3AddressingStyle::Path, max_image_bytes: 1, max_gif_bytes: 1, max_video_bytes: 1, @@ -143,6 +181,35 @@ mod tests { } } + #[test] + fn addressing_style_parses_supported_values() { + assert_eq!( + S3AddressingStyle::from_str("path"), + Ok(S3AddressingStyle::Path) + ); + assert_eq!( + S3AddressingStyle::from_str("virtual"), + Ok(S3AddressingStyle::Virtual) + ); + } + + #[test] + fn addressing_style_defaults_to_path() { + assert_eq!(S3AddressingStyle::default(), S3AddressingStyle::Path); + } + + #[test] + fn addressing_style_rejects_unknown_or_ambiguous_values() { + for invalid in ["", "auto", "PATH", "virtual-hosted"] { + let error = + S3AddressingStyle::from_str(invalid).expect_err("must reject invalid style"); + assert!( + error.contains("BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual'"), + "unexpected error for {invalid:?}: {error}" + ); + } + } + #[test] fn upload_record_knobs_default_off_and_validate() { assert!(valid_config().validate().is_ok()); diff --git a/crates/buzz-media/src/lib.rs b/crates/buzz-media/src/lib.rs index ac05ea6d51..67896d4ef2 100644 --- a/crates/buzz-media/src/lib.rs +++ b/crates/buzz-media/src/lib.rs @@ -17,7 +17,7 @@ pub use bucket_index::{ classify_key, fold_bucket_listing, BucketAggregate, BucketSnapshot, CommunityStorage, KeyClass, Page, SweepError, }; -pub use config::MediaConfig; +pub use config::{MediaConfig, S3AddressingStyle}; pub use error::MediaError; pub use storage::{BlobHeadMeta, BlobMeta, ByteStream, MediaStorage}; pub use types::BlobDescriptor; diff --git a/crates/buzz-media/src/storage.rs b/crates/buzz-media/src/storage.rs index 0e9809af2f..cbf980201f 100644 --- a/crates/buzz-media/src/storage.rs +++ b/crates/buzz-media/src/storage.rs @@ -5,7 +5,7 @@ use std::pin::Pin; use buzz_core::tenant::{CommunityId, TenantContext}; -use crate::config::MediaConfig; +use crate::config::{MediaConfig, S3AddressingStyle}; use crate::error::MediaError; use bytes::Bytes; use s3::creds::Credentials; @@ -61,8 +61,11 @@ impl MediaStorage { } .map_err(|e| MediaError::StorageError(e.to_string()))?; let bucket = Bucket::new(&config.s3_bucket, region, creds) - .map_err(|e| MediaError::StorageError(e.to_string()))? - .with_path_style(); + .map_err(|e| MediaError::StorageError(e.to_string()))?; + let bucket = match config.s3_addressing_style { + S3AddressingStyle::Path => bucket.with_path_style(), + S3AddressingStyle::Virtual => bucket, + }; Ok(Self { bucket }) } @@ -285,6 +288,7 @@ mod tests { s3_secret_key: secret.to_string(), s3_bucket: "buzz-media".to_string(), s3_region: "us-west-2".to_string(), + s3_addressing_style: S3AddressingStyle::Path, max_image_bytes: 50 * 1024 * 1024, max_gif_bytes: 10 * 1024 * 1024, max_video_bytes: 524_288_000, @@ -309,6 +313,23 @@ mod tests { } } + #[test] + fn client_constructor_applies_both_addressing_styles() { + let path = MediaStorage::new(&storage_config("buzz_dev", "buzz_dev_secret")) + .expect("path-style client"); + assert!(path.bucket.is_path_style()); + assert_eq!(path.bucket.url(), "http://localhost:9000/buzz-media"); + + let mut virtual_config = storage_config("buzz_dev", "buzz_dev_secret"); + virtual_config.s3_addressing_style = S3AddressingStyle::Virtual; + let virtual_hosted = MediaStorage::new(&virtual_config).expect("virtual-hosted client"); + assert!(virtual_hosted.bucket.is_subdomain_style()); + assert_eq!( + virtual_hosted.bucket.url(), + "http://buzz-media.localhost:9000" + ); + } + #[test] fn partial_static_keys_are_rejected() { let err = match MediaStorage::new(&storage_config("buzz_dev", "")) { diff --git a/crates/buzz-media/src/upload.rs b/crates/buzz-media/src/upload.rs index 478ac114ef..524b033280 100644 --- a/crates/buzz-media/src/upload.rs +++ b/crates/buzz-media/src/upload.rs @@ -570,6 +570,7 @@ mod tests { s3_secret_key: String::new(), s3_bucket: String::new(), s3_region: "us-east-1".to_string(), + s3_addressing_style: crate::config::S3AddressingStyle::Path, max_image_bytes: 50 * 1024 * 1024, max_gif_bytes: 10 * 1024 * 1024, max_video_bytes: 524_288_000, diff --git a/crates/buzz-media/src/validation.rs b/crates/buzz-media/src/validation.rs index ee940dfb24..f1387fc9d6 100644 --- a/crates/buzz-media/src/validation.rs +++ b/crates/buzz-media/src/validation.rs @@ -949,6 +949,7 @@ mod tests { s3_secret_key: String::new(), s3_bucket: String::new(), s3_region: "us-east-1".to_string(), + s3_addressing_style: crate::config::S3AddressingStyle::Path, max_image_bytes: 50 * 1024 * 1024, max_gif_bytes: 10 * 1024 * 1024, max_video_bytes: 524_288_000, diff --git a/crates/buzz-media/tests/static_creds_minio.rs b/crates/buzz-media/tests/static_creds_minio.rs index d7591238c2..4c8c10702c 100644 --- a/crates/buzz-media/tests/static_creds_minio.rs +++ b/crates/buzz-media/tests/static_creds_minio.rs @@ -1,5 +1,5 @@ -//! Live round-trip test for the **static-credentials** S3 path against a local -//! MinIO, guarded by `#[ignore]`. +//! Live round-trip test for the **static-credentials** S3 path against an +//! S3-compatible service. It is guarded by `#[ignore]`. //! //! This is the path local/dev and any static-key deployment uses //! (`s3_access_key`/`s3_secret_key` both non-empty -> `Credentials::new`). It @@ -15,7 +15,8 @@ //! ``` //! //! Overridable via `BUZZ_S3_ENDPOINT` / `BUZZ_S3_ACCESS_KEY` / -//! `BUZZ_S3_SECRET_KEY` / `BUZZ_S3_BUCKET`. +//! `BUZZ_S3_SECRET_KEY` / `BUZZ_S3_BUCKET` / `BUZZ_S3_REGION` / +//! `BUZZ_S3_ADDRESSING_STYLE`. The default remains `path` for MinIO. use buzz_media::config::MediaConfig; use buzz_media::storage::MediaStorage; @@ -29,7 +30,11 @@ fn minio_config() -> MediaConfig { s3_secret_key: std::env::var("BUZZ_S3_SECRET_KEY") .unwrap_or_else(|_| "buzz_dev_secret".to_string()), s3_bucket: std::env::var("BUZZ_S3_BUCKET").unwrap_or_else(|_| "buzz-media".to_string()), - s3_region: "us-east-1".to_string(), + s3_region: std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()), + s3_addressing_style: std::env::var("BUZZ_S3_ADDRESSING_STYLE") + .unwrap_or_else(|_| "path".to_string()) + .parse() + .expect("BUZZ_S3_ADDRESSING_STYLE must be path or virtual"), max_image_bytes: 50 * 1024 * 1024, max_gif_bytes: 10 * 1024 * 1024, max_video_bytes: 524_288_000, diff --git a/crates/buzz-relay/src/api/git/cas_publish.rs b/crates/buzz-relay/src/api/git/cas_publish.rs index 635dcf2a67..c213e2913e 100644 --- a/crates/buzz-relay/src/api/git/cas_publish.rs +++ b/crates/buzz-relay/src/api/git/cas_publish.rs @@ -1583,22 +1583,15 @@ mod tests { } fn live_store() -> GitStore { - let endpoint = std::env::var("BUZZ_GIT_S3_ENDPOINT") - .or_else(|_| std::env::var("BUZZ_S3_ENDPOINT")) - .unwrap_or_else(|_| "http://localhost:9000".into()); - let access_key = std::env::var("BUZZ_GIT_S3_ACCESS_KEY") - .or_else(|_| std::env::var("BUZZ_S3_ACCESS_KEY")) - .unwrap_or_else(|_| "buzz_dev".into()); - let secret_key = std::env::var("BUZZ_GIT_S3_SECRET_KEY") - .or_else(|_| std::env::var("BUZZ_S3_SECRET_KEY")) - .unwrap_or_else(|_| "buzz_dev_secret".into()); - let bucket = std::env::var("BUZZ_GIT_S3_BUCKET") - .or_else(|_| std::env::var("BUZZ_S3_BUCKET")) - .unwrap_or_else(|_| "buzz-media".into()); - let region = std::env::var("BUZZ_GIT_S3_REGION") - .or_else(|_| std::env::var("BUZZ_S3_REGION")) - .unwrap_or_else(|_| "us-east-1".into()); - GitStore::new(&endpoint, &access_key, &secret_key, &bucket, ®ion).expect("connect minio") + GitStore::new( + "http://localhost:9000", + "buzz_dev", + "buzz_dev_secret", + "buzz-media", + "us-east-1", + buzz_media::config::S3AddressingStyle::Path, + ) + .expect("connect local MinIO") } fn tenant() -> TenantContext { diff --git a/crates/buzz-relay/src/api/git/hydrate.rs b/crates/buzz-relay/src/api/git/hydrate.rs index 064d01923e..3ce809d18f 100644 --- a/crates/buzz-relay/src/api/git/hydrate.rs +++ b/crates/buzz-relay/src/api/git/hydrate.rs @@ -543,8 +543,15 @@ mod tests { #[tokio::test] async fn materialized_repo_is_created_under_configured_scratch_dir() { let scratch = TempDir::new().unwrap(); - let store = GitStore::new("http://localhost:9000", "x", "x", "x", "us-east-1") - .expect("construct store"); + let store = GitStore::new( + "http://localhost:9000", + "x", + "x", + "x", + "us-east-1", + buzz_media::config::S3AddressingStyle::Path, + ) + .expect("construct store"); let manifest = Manifest { version: 1, head: "refs/heads/main".into(), @@ -587,8 +594,9 @@ mod tests { "buzz_dev_secret", "buzz-git", "us-east-1", + buzz_media::config::S3AddressingStyle::Path, ) - .expect("connect minio") + .expect("connect local MinIO") } /// Build a tiny on-disk repo, return (pack bytes, head_oid). diff --git a/crates/buzz-relay/src/api/git/store.rs b/crates/buzz-relay/src/api/git/store.rs index 43d210e648..bdfca8dcf2 100644 --- a/crates/buzz-relay/src/api/git/store.rs +++ b/crates/buzz-relay/src/api/git/store.rs @@ -174,7 +174,9 @@ pub struct GitStore { impl GitStore { /// Build a client against an S3-compatible endpoint (e.g. MinIO). /// - /// Uses path-style addressing for MinIO compatibility; AWS S3 accepts both. + /// `addressing_style` is shared with media storage so both paths sign and + /// route requests consistently. Path style supports the bundled MinIO DNS; + /// virtual-hosted style supports standard S3 and providers such as Railway. /// /// Credential selection mirrors [`buzz_media::MediaStorage::new`]: /// - both `access_key` and `secret_key` non-empty → static credentials @@ -190,6 +192,7 @@ impl GitStore { secret_key: &str, bucket_name: &str, region: &str, + addressing_style: buzz_media::config::S3AddressingStyle, ) -> Result { let region = Region::Custom { region: region.into(), @@ -209,9 +212,11 @@ impl GitStore { } } .map_err(|e| StoreError::Backend(S3Error::Credentials(e)))?; - let bucket = Bucket::new(bucket_name, region, creds) - .map_err(StoreError::Backend)? - .with_path_style(); + let bucket = Bucket::new(bucket_name, region, creds).map_err(StoreError::Backend)?; + let bucket = match addressing_style { + buzz_media::config::S3AddressingStyle::Path => bucket.with_path_style(), + buzz_media::config::S3AddressingStyle::Virtual => bucket, + }; Ok(Self { bucket: Arc::from(bucket), }) @@ -950,6 +955,7 @@ mod tests { "buzz_dev_secret", "buzz-git", "us-west-2", + buzz_media::config::S3AddressingStyle::Path, ) .expect("static creds should build a git store"); match store.bucket.region { @@ -958,6 +964,34 @@ mod tests { } } + #[test] + fn constructor_applies_both_addressing_styles() { + for (style, expected_url, path_style) in [ + ( + buzz_media::config::S3AddressingStyle::Path, + "https://storage.example/buzz-git", + true, + ), + ( + buzz_media::config::S3AddressingStyle::Virtual, + "https://buzz-git.storage.example", + false, + ), + ] { + let store = GitStore::new( + "https://storage.example", + "buzz_dev", + "buzz_dev_secret", + "buzz-git", + "us-east-1", + style, + ) + .expect("construct git store"); + assert_eq!(store.bucket.url(), expected_url); + assert_eq!(store.bucket.is_path_style(), path_style); + } + } + #[test] fn partial_static_keys_are_rejected() { for (access, secret) in [("buzz_dev", ""), ("", "buzz_dev_secret")] { @@ -967,6 +1001,7 @@ mod tests { secret, "buzz-git", "us-east-1", + buzz_media::config::S3AddressingStyle::Path, ) { Ok(_) => { panic!("partial static creds must not silently use the credential chain") @@ -998,14 +1033,29 @@ mod probe { } fn store() -> GitStore { + // This is the dedicated backend conformance path, so all connection and + // signing inputs are overridable for a real provider such as Railway. + // The hydrate/CAS live tests use explicit local MinIO fixtures instead. + let endpoint = + std::env::var("BUZZ_S3_ENDPOINT").unwrap_or_else(|_| "http://localhost:9000".into()); + let access_key = std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_else(|_| "buzz_dev".into()); + let secret_key = + std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_else(|_| "buzz_dev_secret".into()); + let bucket = std::env::var("BUZZ_S3_BUCKET").unwrap_or_else(|_| "buzz-git".into()); + let region = std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".into()); + let addressing_style = std::env::var("BUZZ_S3_ADDRESSING_STYLE") + .unwrap_or_else(|_| "path".into()) + .parse() + .expect("BUZZ_S3_ADDRESSING_STYLE must be path or virtual"); GitStore::new( - "http://localhost:9000", - "buzz_dev", - "buzz_dev_secret", - "buzz-git", - "us-east-1", + &endpoint, + &access_key, + &secret_key, + &bucket, + ®ion, + addressing_style, ) - .expect("connect minio") + .expect("connect S3-compatible storage") } fn sha256_hex(b: &[u8]) -> String { diff --git a/crates/buzz-relay/src/config.rs b/crates/buzz-relay/src/config.rs index a1691349d6..e494355736 100644 --- a/crates/buzz-relay/src/config.rs +++ b/crates/buzz-relay/src/config.rs @@ -630,6 +630,16 @@ impl Config { .and_then(|v| v.parse().ok()) .unwrap_or(9102); + let s3_addressing_style = match std::env::var("BUZZ_S3_ADDRESSING_STYLE") { + Ok(value) => value.parse().map_err(ConfigError::InvalidValue)?, + Err(std::env::VarError::NotPresent) => buzz_media::config::S3AddressingStyle::default(), + Err(std::env::VarError::NotUnicode(_)) => { + return Err(ConfigError::InvalidValue( + "BUZZ_S3_ADDRESSING_STYLE must be valid Unicode and one of 'path' or 'virtual'" + .to_string(), + )); + } + }; let media = buzz_media::MediaConfig { s3_endpoint: std::env::var("BUZZ_S3_ENDPOINT") .unwrap_or_else(|_| "http://localhost:9000".to_string()), @@ -641,6 +651,7 @@ impl Config { s3_region: std::env::var("BUZZ_S3_REGION") .or_else(|_| std::env::var("AWS_REGION")) .unwrap_or_else(|_| "us-east-1".to_string()), + s3_addressing_style, max_image_bytes: std::env::var("BUZZ_MAX_IMAGE_BYTES") .ok() .and_then(|v| v.parse().ok()) @@ -990,6 +1001,11 @@ mod tests { !config.require_media_get_auth, "require_media_get_auth should default to false for staged client rollout" ); + assert_eq!( + config.media.s3_addressing_style, + buzz_media::config::S3AddressingStyle::Path, + "S3 addressing must default to path style for bundled MinIO compatibility" + ); assert!( config.join_policy.is_none(), "join_policy should default to None so policy prompts and acceptance receipts are opt-in" @@ -1000,6 +1016,61 @@ mod tests { ); } + #[test] + fn s3_addressing_style_env_accepts_virtual_and_rejects_invalid_values() { + let _guard = ENV_MUTEX.lock().unwrap(); + let previous = std::env::var_os("BUZZ_S3_ADDRESSING_STYLE"); + + std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", "virtual"); + let configured = Config::from_env() + .expect("virtual style config") + .media + .s3_addressing_style; + + std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", "auto"); + let invalid = Config::from_env(); + + if let Some(value) = previous { + std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", value); + } else { + std::env::remove_var("BUZZ_S3_ADDRESSING_STYLE"); + } + + assert_eq!(configured, buzz_media::config::S3AddressingStyle::Virtual); + assert!(matches!( + invalid, + Err(ConfigError::InvalidValue(ref message)) + if message.contains("BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual'") + )); + } + + #[cfg(unix)] + #[test] + fn s3_addressing_style_env_rejects_non_unicode_values() { + use std::os::unix::ffi::OsStringExt; + + let _guard = ENV_MUTEX.lock().unwrap(); + let previous = std::env::var_os("BUZZ_S3_ADDRESSING_STYLE"); + std::env::set_var( + "BUZZ_S3_ADDRESSING_STYLE", + std::ffi::OsString::from_vec(vec![0xff]), + ); + + let invalid = Config::from_env(); + + if let Some(value) = previous { + std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", value); + } else { + std::env::remove_var("BUZZ_S3_ADDRESSING_STYLE"); + } + + assert!(matches!( + invalid, + Err(ConfigError::InvalidValue(ref message)) + if message.contains("must be valid Unicode") + )); + } + #[test] fn redis_pool_size_env_override_and_invalid_fallback() { let _guard = ENV_MUTEX.lock().unwrap(); diff --git a/crates/buzz-relay/src/state.rs b/crates/buzz-relay/src/state.rs index 758c001b96..58a869a995 100644 --- a/crates/buzz-relay/src/state.rs +++ b/crates/buzz-relay/src/state.rs @@ -697,6 +697,7 @@ impl AppState { &config.media.s3_secret_key, &config.media.s3_bucket, &config.media.s3_region, + config.media.s3_addressing_style, ) .expect("media storage was already constructed with this S3 config"); let git_pack_cache = Arc::new( diff --git a/crates/buzz-test-client/Cargo.toml b/crates/buzz-test-client/Cargo.toml index 40a08f3d19..e495c16300 100644 --- a/crates/buzz-test-client/Cargo.toml +++ b/crates/buzz-test-client/Cargo.toml @@ -36,6 +36,7 @@ sha2 = { workspace = true } sqlx = { workspace = true } chrono = { workspace = true } s3 = { version = "0.37", package = "rust-s3", default-features = false, features = ["tokio-rustls-tls", "fail-on-err", "tags"] } +buzz-media = { workspace = true } buzz-sdk = { workspace = true } [[bin]] diff --git a/crates/buzz-test-client/tests/e2e_git.rs b/crates/buzz-test-client/tests/e2e_git.rs index 63281fd18f..f543e56229 100644 --- a/crates/buzz-test-client/tests/e2e_git.rs +++ b/crates/buzz-test-client/tests/e2e_git.rs @@ -21,6 +21,7 @@ use std::path::{Path, PathBuf}; use std::process::Command; use std::time::Duration; +use buzz_media::S3AddressingStyle; use nostr::{EventBuilder, Keys, Kind, Tag}; use s3::creds::Credentials; use s3::{Bucket, Region}; @@ -115,29 +116,55 @@ struct PointerSnapshot { } impl GitS3Probe { - fn from_env() -> Self { - let endpoint = std::env::var("BUZZ_GIT_S3_ENDPOINT") - .or_else(|_| std::env::var("BUZZ_S3_ENDPOINT")) - .unwrap_or_else(|_| "http://localhost:9000".to_string()); - let access_key = std::env::var("BUZZ_GIT_S3_ACCESS_KEY") - .or_else(|_| std::env::var("BUZZ_S3_ACCESS_KEY")) - .unwrap_or_else(|_| "buzz_dev".to_string()); - let secret_key = std::env::var("BUZZ_GIT_S3_SECRET_KEY") - .or_else(|_| std::env::var("BUZZ_S3_SECRET_KEY")) - .unwrap_or_else(|_| "buzz_dev_secret".to_string()); - let bucket = std::env::var("BUZZ_GIT_S3_BUCKET") - .or_else(|_| std::env::var("BUZZ_S3_BUCKET")) - .unwrap_or_else(|_| "buzz-media".to_string()); - + fn bucket( + endpoint: String, + access_key: &str, + secret_key: &str, + bucket_name: &str, + region_name: String, + addressing_style: S3AddressingStyle, + ) -> Box { let region = Region::Custom { - region: "us-east-1".into(), + region: region_name, endpoint, }; - let creds = Credentials::new(Some(&access_key), Some(&secret_key), None, None, None) + let creds = Credentials::new(Some(access_key), Some(secret_key), None, None, None) .expect("S3 credentials"); - let bucket = Bucket::new(&bucket, region, creds) - .expect("S3 bucket") - .with_path_style(); + let bucket = Bucket::new(bucket_name, region, creds).expect("S3 bucket"); + match addressing_style { + S3AddressingStyle::Path => bucket.with_path_style(), + S3AddressingStyle::Virtual => bucket, + } + } + + fn from_env() -> Self { + // These E2E assertions inspect the relay's backing bucket directly, so + // they must receive the same provider connection and URL style as the + // relay. Unit/live MinIO probes in buzz-relay keep explicit local + // fixtures and do not need provider overrides. + let endpoint = std::env::var("BUZZ_S3_ENDPOINT") + .unwrap_or_else(|_| "http://localhost:9000".to_string()); + let access_key = + std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_else(|_| "buzz_dev".to_string()); + let secret_key = + std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_else(|_| "buzz_dev_secret".to_string()); + let bucket_name = + std::env::var("BUZZ_S3_BUCKET").unwrap_or_else(|_| "buzz-media".to_string()); + let region_name = + std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()); + let addressing_style = std::env::var("BUZZ_S3_ADDRESSING_STYLE") + .unwrap_or_else(|_| "path".to_string()) + .parse::() + .expect("BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual'"); + + let bucket = Self::bucket( + endpoint, + &access_key, + &secret_key, + &bucket_name, + region_name, + addressing_style, + ); Self { bucket } } @@ -192,6 +219,31 @@ impl GitS3Probe { } } +#[test] +fn git_s3_probe_builds_both_addressing_styles() { + let path = GitS3Probe::bucket( + "https://storage.example".to_string(), + "access", + "secret", + "buzz-media", + "us-east-1".to_string(), + S3AddressingStyle::Path, + ); + assert!(path.is_path_style()); + assert_eq!(path.url(), "https://storage.example/buzz-media"); + + let virtual_hosted = GitS3Probe::bucket( + "https://storage.example".to_string(), + "access", + "secret", + "buzz-media", + "auto".to_string(), + S3AddressingStyle::Virtual, + ); + assert!(virtual_hosted.is_subdomain_style()); + assert_eq!(virtual_hosted.url(), "https://buzz-media.storage.example"); +} + #[tokio::test] #[ignore = "requires live relay + MinIO + git"] async fn git_clone_push_fetch_force_roundtrip() { diff --git a/deploy/charts/buzz/README.md b/deploy/charts/buzz/README.md index a7c4bcf63b..b2778df28b 100644 --- a/deploy/charts/buzz/README.md +++ b/deploy/charts/buzz/README.md @@ -52,6 +52,48 @@ See: The chart fails at `helm install` / `helm template` time with a clear message if any of these are missing or malformed (see `templates/_validate.tpl`). +## S3 URL addressing + +Buzz uses one URL style for both media and Git/CAS object-store requests: + +| `s3.addressingStyle` | Request shape | Use for | +|---|---|---| +| `path` (default) | `https://endpoint/bucket/key` | Bundled MinIO and endpoints whose DNS does not resolve bucket subdomains | +| `virtual` | `https://bucket.endpoint/key` | AWS-style providers and new Railway Storage Buckets | + +The chart always renders `s3.addressingStyle` as +`BUZZ_S3_ADDRESSING_STYLE`. It renders `s3.region` as `BUZZ_S3_REGION` only +when explicitly set, preserving the relay's existing `AWS_REGION` fallback for +upgrades. Only `path` and `virtual` addressing styles are accepted; invalid +values fail chart rendering and relay startup. The bundled MinIO quickstart +deliberately keeps `path` because its Service DNS resolves one endpoint +hostname, not arbitrary `.` names. + +For a Railway Storage Bucket, map its variables to chart values in the service +or generated Helm configuration: + +```yaml +s3: + endpoint: "${{Object Storage.ENDPOINT}}" + bucket: "${{Object Storage.BUCKET}}" + region: "${{Object Storage.REGION}}" + addressingStyle: virtual +``` + +Store `BUZZ_S3_ACCESS_KEY=${{Object Storage.ACCESS_KEY_ID}}` and +`BUZZ_S3_SECRET_KEY=${{Object Storage.SECRET_ACCESS_KEY}}` in the Secret named by +`secrets.existingSecret`. Railway's Credentials tab is authoritative for older +buckets, which may still require `path`. The setting changes request routing and +SigV4 signing, so do not put the bucket into `s3.endpoint`; pass Railway's base +`ENDPOINT` and `BUCKET` separately. + +Object storage is contacted during relay startup only when +`BUZZ_GIT_CONFORMANCE_PROBE` is enabled (the relay default). A probe failure is +startup-fatal, so Kubernetes readiness never opens. If an operator explicitly +disables that probe through `relay.extraEnv`, `/_readiness` does not test object +storage; configuration is still parsed strictly, but reachability and addressing +errors surface on the first storage operation. + ## Relay Pod extensions The chart exposes narrow extension points for init containers, volumes, relay diff --git a/deploy/charts/buzz/examples/argocd-app.yaml b/deploy/charts/buzz/examples/argocd-app.yaml index 7e90f64bd7..8f6cb76228 100644 --- a/deploy/charts/buzz/examples/argocd-app.yaml +++ b/deploy/charts/buzz/examples/argocd-app.yaml @@ -41,6 +41,8 @@ spec: s3: endpoint: "https://s3.us-east-1.amazonaws.com" bucket: "buzz-media" + region: "us-east-1" + addressingStyle: virtual # accessKey / secretKey live in buzz-secrets persistence: diff --git a/deploy/charts/buzz/examples/flux-helmrelease.yaml b/deploy/charts/buzz/examples/flux-helmrelease.yaml index 16754c0fcb..09a6bfeb6a 100644 --- a/deploy/charts/buzz/examples/flux-helmrelease.yaml +++ b/deploy/charts/buzz/examples/flux-helmrelease.yaml @@ -41,6 +41,8 @@ spec: s3: endpoint: "https://s3.us-east-1.amazonaws.com" bucket: "buzz-media" + region: "us-east-1" + addressingStyle: virtual persistence: git: diff --git a/deploy/charts/buzz/templates/_validate.tpl b/deploy/charts/buzz/templates/_validate.tpl index 946424f9a3..aa7f7ac13c 100644 --- a/deploy/charts/buzz/templates/_validate.tpl +++ b/deploy/charts/buzz/templates/_validate.tpl @@ -75,10 +75,12 @@ surface at template time regardless of which manifest helm renders first. {{- fail "Postgres source missing: enable postgresql.enabled=true, set externalPostgresql.url, or provide secrets.existingSecret with key DATABASE_URL." -}} {{- end -}} -{{/* S3 / object-storage source must exist somewhere (relay hard-fails its - startup conformance probe without a reachable bucket). */}} +{{/* S3 / object-storage source must exist somewhere. With the default + BUZZ_GIT_CONFORMANCE_PROBE behavior, an unreachable bucket is detected + before the relay opens its listener; operators can explicitly disable that + startup gate. */}} {{- if not (or .Values.minio.enabled .Values.s3.endpoint .Values.secrets.existingSecret) -}} - {{- fail "S3/object-storage source missing: enable minio.enabled=true (quickstart in-cluster), set s3.endpoint + s3.bucket + credentials, or provide secrets.existingSecret with keys BUZZ_S3_ACCESS_KEY + BUZZ_S3_SECRET_KEY. The relay runs a startup S3 conformance probe and exits if storage is unreachable." -}} + {{- fail "S3/object-storage source missing: enable minio.enabled=true (quickstart in-cluster), set s3.endpoint + s3.bucket + credentials, or provide secrets.existingSecret with keys BUZZ_S3_ACCESS_KEY + BUZZ_S3_SECRET_KEY. By default the relay runs a startup S3 conformance probe and exits if storage is unreachable; disabling BUZZ_GIT_CONFORMANCE_PROBE also removes that startup storage check." -}} {{- end -}} {{- end -}} diff --git a/deploy/charts/buzz/templates/deployment.yaml b/deploy/charts/buzz/templates/deployment.yaml index bf2df4c2c8..67a93138c5 100644 --- a/deploy/charts/buzz/templates/deployment.yaml +++ b/deploy/charts/buzz/templates/deployment.yaml @@ -170,6 +170,10 @@ spec: - { name: BUZZ_S3_ENDPOINT, value: {{ $s3Endpoint | quote }} } {{- end }} - { name: BUZZ_S3_BUCKET, value: {{ .Values.s3.bucket | quote }} } + {{- if .Values.s3.region }} + - { name: BUZZ_S3_REGION, value: {{ .Values.s3.region | quote }} } + {{- end }} + - { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ .Values.s3.addressingStyle | quote }} } # ── Secrets (from chart-managed or existing) ───────────── - name: BUZZ_RELAY_PRIVATE_KEY diff --git a/deploy/charts/buzz/tests/render_test.yaml b/deploy/charts/buzz/tests/render_test.yaml index 3e044f5d7c..cf08210781 100644 --- a/deploy/charts/buzz/tests/render_test.yaml +++ b/deploy/charts/buzz/tests/render_test.yaml @@ -30,6 +30,18 @@ tests: path: kind value: Service template: templates/service.yaml + - notContains: + path: spec.template.spec.containers[0].env + content: + name: BUZZ_S3_REGION + any: true + template: templates/deployment.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: BUZZ_S3_ADDRESSING_STYLE + value: "path" + template: templates/deployment.yaml - contains: path: spec.template.spec.containers[0].env content: @@ -47,6 +59,32 @@ tests: value: "true" template: templates/deployment.yaml + - it: renders virtual-hosted S3 addressing for providers that require it + set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + externalPostgresql.url: postgres://u:p@h:5432/d + externalRedis.url: redis://h:6379 + s3.endpoint: https://storage.railway.app + s3.bucket: buzz-media-example + s3.region: auto + s3.addressingStyle: virtual + s3.accessKey: a + s3.secretKey: s + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BUZZ_S3_REGION + value: "auto" + template: templates/deployment.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: BUZZ_S3_ADDRESSING_STYLE + value: "virtual" + template: templates/deployment.yaml + - it: lets an explicit value opt out of media read auth for dev/public deployments set: relayUrl: wss://buzz.example.com diff --git a/deploy/charts/buzz/tests/validation_test.yaml b/deploy/charts/buzz/tests/validation_test.yaml index f0a3869795..a5a0050a86 100644 --- a/deploy/charts/buzz/tests/validation_test.yaml +++ b/deploy/charts/buzz/tests/validation_test.yaml @@ -58,6 +58,17 @@ tests: - failedTemplate: errorPattern: "Postgres source missing" + - it: rejects an invalid S3 addressing style + set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + externalPostgresql.url: postgres://u:p@h:5432/d + s3.endpoint: http://minio:9000 + s3.addressingStyle: auto + asserts: + - failedTemplate: + errorPattern: "s3.addressingStyle: s3.addressingStyle must be one of the following:.*path.*virtual" + - it: fails when S3/object-storage source is missing set: relayUrl: wss://buzz.example.com diff --git a/deploy/charts/buzz/values.schema.json b/deploy/charts/buzz/values.schema.json index 53bb29bb60..9cb6a02c9b 100644 --- a/deploy/charts/buzz/values.schema.json +++ b/deploy/charts/buzz/values.schema.json @@ -198,6 +198,15 @@ "properties": { "endpoint": { "type": "string", "pattern": "^(https?://.+)?$" }, "bucket": { "type": "string", "minLength": 1 }, + "region": { + "type": "string", + "description": "Optional S3 region used for SigV4 signing. When empty, BUZZ_S3_REGION is omitted so the relay can use AWS_REGION or its own default." + }, + "addressingStyle": { + "type": "string", + "enum": ["path", "virtual"], + "description": "S3 URL style shared by media and Git/CAS clients. Defaults to path for bundled MinIO compatibility." + }, "accessKey": { "type": "string" }, "secretKey": { "type": "string" } } diff --git a/deploy/charts/buzz/values.yaml b/deploy/charts/buzz/values.yaml index 8ac5086e27..810f8a9658 100644 --- a/deploy/charts/buzz/values.yaml +++ b/deploy/charts/buzz/values.yaml @@ -338,6 +338,12 @@ externalRedis: s3: endpoint: "" bucket: "buzz-media" + # Optional SigV4 signing region. Leave empty to preserve the relay's + # AWS_REGION fallback; set the provider's credential value when needed. + region: "" + # path: https://endpoint/bucket/key (bundled MinIO-compatible default) + # virtual: https://bucket.endpoint/key (standard S3; required by new Railway buckets) + addressingStyle: path accessKey: "" secretKey: "" diff --git a/deploy/compose/.env.example b/deploy/compose/.env.example index 838824c17f..f6ab4fcab9 100644 --- a/deploy/compose/.env.example +++ b/deploy/compose/.env.example @@ -33,6 +33,8 @@ REDIS_PASSWORD=CHANGE_ME_RANDOM_PASSWORD BUZZ_S3_ACCESS_KEY=CHANGE_ME_RANDOM_ACCESS_KEY BUZZ_S3_SECRET_KEY=CHANGE_ME_RANDOM_SECRET_KEY BUZZ_S3_BUCKET=buzz-media +# Bundled MinIO uses path-style URLs; deploy/compose/compose.yml pins this. +BUZZ_S3_ADDRESSING_STYLE=path # Optional host ports. Base compose publishes the relay directly on BUZZ_HTTP_PORT. BUZZ_HTTP_PORT=3000 diff --git a/deploy/compose/README.md b/deploy/compose/README.md index 0de524fb5b..bb0e63fe15 100644 --- a/deploy/compose/README.md +++ b/deploy/compose/README.md @@ -38,6 +38,11 @@ keypair. migrations. - The stack uses Postgres, Redis, MinIO, and a git data volume because those are real Buzz dependencies today. Minimal mode can simplify this later. +- The bundled Compose stack fixes the relay endpoint to `http://minio:9000` and + `BUZZ_S3_ADDRESSING_STYLE=path`: Docker DNS resolves `minio`, not + `.minio`. It is not configurable for an external S3 provider through + `.env`; use the Helm chart or a custom Compose configuration for providers + such as new Railway Storage Buckets that require `virtual` addressing. Run `./run.sh backup-hint` for the backup checklist. diff --git a/deploy/compose/compose.yml b/deploy/compose/compose.yml index bc3c27501e..15337c92a2 100644 --- a/deploy/compose/compose.yml +++ b/deploy/compose/compose.yml @@ -12,6 +12,8 @@ services: DATABASE_URL: postgres://${POSTGRES_USER:-buzz}:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-buzz} REDIS_URL: redis://:${REDIS_PASSWORD:?set REDIS_PASSWORD}@redis:6379 BUZZ_S3_ENDPOINT: http://minio:9000 + # Docker DNS resolves `minio`, not arbitrary `.minio` hosts. + BUZZ_S3_ADDRESSING_STYLE: path BUZZ_S3_ACCESS_KEY: ${BUZZ_S3_ACCESS_KEY:?set BUZZ_S3_ACCESS_KEY} BUZZ_S3_SECRET_KEY: ${BUZZ_S3_SECRET_KEY:?set BUZZ_S3_SECRET_KEY} BUZZ_S3_BUCKET: ${BUZZ_S3_BUCKET:-buzz-media} diff --git a/desktop/src-tauri/src/commands/media_snapshot_png.rs b/desktop/src-tauri/src/commands/media_snapshot_png.rs index 734d8f5dc8..bcaec6a592 100644 --- a/desktop/src-tauri/src/commands/media_snapshot_png.rs +++ b/desktop/src-tauri/src/commands/media_snapshot_png.rs @@ -204,6 +204,7 @@ mod tests { s3_secret_key: String::new(), s3_bucket: String::new(), s3_region: "us-east-1".to_string(), + s3_addressing_style: buzz_media_pkg::S3AddressingStyle::Path, max_image_bytes: 50 * 1024 * 1024, max_gif_bytes: 10 * 1024 * 1024, max_video_bytes: 524_288_000, From 788b3c002bd2509455444f57f8a03a054b4b496a Mon Sep 17 00:00:00 2001 From: Tyler <109685178+tlongwell-block@users.noreply.github.com> Date: Wed, 29 Jul 2026 20:07:55 -0400 Subject: [PATCH 29/32] fix(git): channel binding tooling + author remediation for unbound repos (#3626) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #3527. Repos announced via vanilla NIP-34 (kind:30617 without a `buzz-channel` tag) 404 forever: the SEC-005 read gate requires a channel-membership ACL, and nothing tells the author why or how to fix it. Per the ruling in the originating thread, this ships **bind/rebind tooling plus a narrow author-only remediation carve-out** — the shelved owner-circle approach is intentionally absent. ## Relay - **`api/git/binding.rs` (new):** shared tri-state binding resolver — `Bound(uuid)` / `NotBound` / `Broken`. First-tag, fail-closed: a malformed `buzz-channel` tag is `Broken`, never conflated with "no tag". Both gates use it. - **Read gate (`transport.rs`):** a **never-bound** repo read by **its own announcement author** still returns 404 (status byte-identical to the generic denial) but the body carries remediation: `run: buzz repos bind --id --channel — …`. This leaks nothing — the author announced the repo, and only the author can rebind (30617 is keyed by `(author, d)`). `Broken` bindings stay generic-denial for everyone, including the author (revocation shape). Bound-to-nonexistent-channel stays generic (phase 1; ingest validation is phase 2). - **Push gate (`policy.rs`):** unbound denial now returns `GIT_NO_CHANNEL_BINDING_BODY`. A deploy-skew test pins that the body carries both the new token (`no_channel_binding`) and the legacy phrase (`"no channel binding"`) so already-shipped desktops keep matching. **(Review r1, blocker 2)** `Broken` no longer collapses into "unbound": it denies 403 `invalid channel binding` for *everyone — including the announcement owner —* **before** the owner short-circuit, matching the read gate's fail-closed posture. The remediation token stays NotBound-only. - **`ingest.rs`:** side-effect failure `warn!` → `error!` — prod runs `RUST_LOG=error`, so these failures were invisible during triage. ## Contract - **`buzz-core/git_perms.rs`:** `GIT_NO_CHANNEL_BINDING_TOKEN` / `GIT_NO_CHANNEL_BINDING_BODY` consts as the declared cross-component contract; relay tests and desktop matcher both build on them. ## CLI - **`buzz repos bind --id --channel `** — rebinds an existing announcement, preserving other tags. - **(Review r1, blocker 1)** **`--channel` on `buzz repos create`** — optional; injects exactly one shape-validated `buzz-channel` tag at creation via a pure `build_create_announcement` builder, so the primary create command stops producing repos the relay 404s. UUID existence/membership stays the relay's authority at git-access time (same TOCTOU posture as `repos bind`). Overlaps with #3594 (open, head 6bbe38459) — happy to reconcile whichever lands first; this branch also carries the bind path and tag preservation. ## Desktop - **Rust:** new `commands/project_git_merge_error.rs` (extracted from `project_git_workflow.rs` to respect the 1000-line ratchet); maps the token to a structured `no_channel_binding` error carrying the bind command. - **TS:** new `features/projects/lib/projectBranchErrors.ts` + tests — dual matcher (new token AND legacy spaced phrase); `ProjectBranchDialogs.tsx` uses it. ## Tests / verification (at head f914c7066, base 581baa625) - Workspace `cargo test` green; `clippy -D warnings` clean; desktop Rust 1859 pass; TS 3780 pass; tsc/biome/file-size checks pass. Pre-push hooks re-ran all suites at the pushed head. - Postgres-gated `sec005_read_gate_tests`: all 6 pass, including `read_gate_gives_author_of_unbound_repo_remediation_body` — asserts 404 status, `text/plain` content-type, and exact body bytes, distinguishing remediation from generic denial (a blind `is_err()` can't). - **New (review r1):** `buzz-cli` emitted-event tests — `create_with_channel_emits_exactly_one_binding_tag`, `create_without_channel_emits_no_binding_tag`, `create_rejects_malformed_channel_uuid` (266/266 pass). Postgres-gated `push_gate_denies_owner_through_broken_binding` — owner + malformed-first/valid-second binding → 403 generic body without the remediation token; never-bound control stays 200, pinning the denial to `Broken` specifically. - e2e git tests now bind announcements to a real channel via a `create_test_channel` helper. --------- Signed-off-by: Tyler Longwell Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell --- crates/buzz-cli/src/commands/repos.rs | 257 +++++++++++++++-- crates/buzz-cli/src/lib.rs | 23 +- crates/buzz-core/src/git_perms.rs | 24 ++ crates/buzz-relay/src/api/git/binding.rs | 128 +++++++++ crates/buzz-relay/src/api/git/mod.rs | 1 + crates/buzz-relay/src/api/git/policy.rs | 228 ++++++++++++++- crates/buzz-relay/src/api/git/transport.rs | 269 +++++++++++------- crates/buzz-relay/src/handlers/ingest.rs | 7 +- crates/buzz-test-client/tests/e2e_git.rs | 28 ++ desktop/src-tauri/src/commands/mod.rs | 1 + .../src/commands/project_git_merge_error.rs | 152 ++++++++++ .../src/commands/project_git_workflow.rs | 114 +------- .../projects/lib/projectBranchErrors.test.mjs | 43 +++ .../projects/lib/projectBranchErrors.ts | 35 +++ .../projects/ui/ProjectBranchDialogs.tsx | 17 +- 15 files changed, 1069 insertions(+), 258 deletions(-) create mode 100644 crates/buzz-relay/src/api/git/binding.rs create mode 100644 desktop/src-tauri/src/commands/project_git_merge_error.rs create mode 100644 desktop/src/features/projects/lib/projectBranchErrors.test.mjs create mode 100644 desktop/src/features/projects/lib/projectBranchErrors.ts diff --git a/crates/buzz-cli/src/commands/repos.rs b/crates/buzz-cli/src/commands/repos.rs index 0f570df1aa..608d495055 100644 --- a/crates/buzz-cli/src/commands/repos.rs +++ b/crates/buzz-cli/src/commands/repos.rs @@ -83,27 +83,36 @@ fn build_protection_tag( Tag::parse(values).map_err(tag_error) } -enum ProtectionChange { - Set(Box), - Remove(String), +enum RepoChange { + SetProtection(Box), + RemoveProtection(String), + /// Bind (or rebind) the repo to a channel: replaces every existing + /// `buzz-channel` tag with exactly one carrying the validated UUID. + BindChannel(String), } fn build_updated_repo_announcement( existing: &Event, - change: ProtectionChange, + change: RepoChange, ) -> Result { let repo_id = repo_id_from_event(existing)?; - let (pattern, replacement) = match change { - ProtectionChange::Set(tag) => { + // What to strip beyond `auth` (always stripped), and what to append. + let (removed_pattern, removed_channel, replacement) = match change { + RepoChange::SetProtection(tag) => { let pattern = protection_pattern(&tag) .ok_or_else(|| CliError::Other("replacement is not a protection tag".into()))? .to_string(); - (pattern, Some(*tag)) + (Some(pattern), false, Some(*tag)) } - ProtectionChange::Remove(pattern) => { + RepoChange::RemoveProtection(pattern) => { RefPattern::parse(&pattern) .map_err(|error| CliError::Usage(format!("invalid ref pattern: {error}")))?; - (pattern, None) + (Some(pattern), false, None) + } + RepoChange::BindChannel(channel) => { + crate::validate::validate_uuid(&channel)?; + let tag = Tag::parse(["buzz-channel", channel.as_str()]).map_err(tag_error)?; + (None, true, Some(tag)) } }; @@ -111,7 +120,13 @@ fn build_updated_repo_announcement( .tags .iter() .filter(|tag| { - !has_tag_name(tag, "auth") && protection_pattern(tag) != Some(pattern.as_str()) + if has_tag_name(tag, "auth") { + return false; + } + if removed_channel && has_tag_name(tag, "buzz-channel") { + return false; + } + removed_pattern.is_none() || protection_pattern(tag) != removed_pattern.as_deref() }) .cloned() .collect(); @@ -199,21 +214,30 @@ async fn submit_repo_update(client: &BuzzClient, builder: EventBuilder) -> Resul Ok(()) } -pub async fn cmd_create_repo( - client: &BuzzClient, +/// Build the kind:30617 announcement for `repos create`, including the +/// `buzz-channel` binding when requested. +/// +/// Pure (no I/O) so the emitted tags are unit-testable. Exactly one +/// validated `buzz-channel` tag is appended — the tag is the git ACL +/// (issue #3527: without it the relay 404s every clone/fetch/push), so the +/// UUID is shape-validated here and its existence/membership is the relay's +/// authority at git-access time, same posture as `repos bind`. +#[allow(clippy::too_many_arguments)] +fn build_create_announcement( repo_id: &str, name: Option<&str>, description: Option<&str>, clone_urls: &[String], web_url: Option<&str>, relays: &[String], -) -> Result<(), CliError> { + channel: Option<&str>, +) -> Result { validate_repo_id(repo_id)?; let clone_refs: Vec<&str> = clone_urls.iter().map(|s| s.as_str()).collect(); let relay_refs: Vec<&str> = relays.iter().map(|s| s.as_str()).collect(); - let builder = buzz_sdk::build_repo_announcement( + let mut builder = buzz_sdk::build_repo_announcement( repo_id, name, description, @@ -223,6 +247,33 @@ pub async fn cmd_create_repo( ) .map_err(|e| CliError::Other(format!("build_repo_announcement failed: {e}")))?; + if let Some(channel) = channel { + crate::validate::validate_uuid(channel)?; + builder = builder.tag(Tag::parse(["buzz-channel", channel]).map_err(tag_error)?); + } + Ok(builder) +} + +#[allow(clippy::too_many_arguments)] +pub async fn cmd_create_repo( + client: &BuzzClient, + repo_id: &str, + name: Option<&str>, + description: Option<&str>, + clone_urls: &[String], + web_url: Option<&str>, + relays: &[String], + channel: Option<&str>, +) -> Result<(), CliError> { + let builder = build_create_announcement( + repo_id, + name, + description, + clone_urls, + web_url, + relays, + channel, + )?; let event = client.sign_event(builder)?; let resp = client.submit_event(event).await?; println!("{resp}"); @@ -320,7 +371,8 @@ async fn cmd_protect_set( require_patch, )?; let event = current_repo(client, repo_id).await?; - let builder = build_updated_repo_announcement(&event, ProtectionChange::Set(Box::new(tag)))?; + let builder = + build_updated_repo_announcement(&event, RepoChange::SetProtection(Box::new(tag)))?; submit_repo_update(client, builder).await } @@ -341,8 +393,27 @@ async fn cmd_protect_remove( "repository {repo_id:?} has no protection rule for {ref_pattern:?}" ))); } + let builder = build_updated_repo_announcement( + &event, + RepoChange::RemoveProtection(ref_pattern.to_string()), + )?; + submit_repo_update(client, builder).await +} + +/// Bind (or rebind) a repository to a channel — the fix path for issue +/// #3527's permanently-404 repos. Publishes a read-modify-write update of +/// the caller's own kind:30617 with exactly one `buzz-channel` tag; all +/// other metadata (protections, name, description, future tags) is +/// preserved by the same machinery `repos protect` uses. +/// +/// The UUID is validated for *shape* only — deliberately. Channel existence +/// and the caller's membership are the relay's authority at git-access +/// time; a CLI-side network pre-check would just be TOCTOU with extra +/// latency. +async fn cmd_bind_repo(client: &BuzzClient, repo_id: &str, channel: &str) -> Result<(), CliError> { + let event = current_repo(client, repo_id).await?; let builder = - build_updated_repo_announcement(&event, ProtectionChange::Remove(ref_pattern.to_string()))?; + build_updated_repo_announcement(&event, RepoChange::BindChannel(channel.to_string()))?; submit_repo_update(client, builder).await } @@ -356,6 +427,7 @@ pub async fn dispatch(cmd: crate::ReposCmd, client: &BuzzClient) -> Result<(), C clone_urls, web, relays, + channel, } => { cmd_create_repo( client, @@ -365,11 +437,13 @@ pub async fn dispatch(cmd: crate::ReposCmd, client: &BuzzClient) -> Result<(), C &clone_urls, web.as_deref(), &relays, + channel.as_deref(), ) .await } ReposCmd::Get { id, owner } => cmd_get_repo(client, &id, owner.as_deref()).await, ReposCmd::List { owner, limit } => cmd_list_repos(client, owner.as_deref(), limit).await, + ReposCmd::Bind { id, channel } => cmd_bind_repo(client, &id, &channel).await, ReposCmd::Protect(command) => match command { ReposProtectCmd::List { id } => cmd_protect_list(client, &id).await, ReposProtectCmd::Set { @@ -403,8 +477,8 @@ mod tests { use nostr::{EventBuilder, Keys, Kind, Tag, Timestamp}; use super::{ - build_protection_tag, build_updated_repo_announcement, protection_rules_json, - validate_write_response, ProtectionChange, + build_create_announcement, build_protection_tag, build_updated_repo_announcement, + protection_rules_json, validate_write_response, RepoChange, }; fn signed_repo(tags: Vec, content: &str, created_at: u64) -> nostr::Event { @@ -439,7 +513,7 @@ mod tests { let updated = build_updated_repo_announcement( &existing, - ProtectionChange::Set(Box::new(replacement)), + RepoChange::SetProtection(Box::new(replacement)), ) .expect("build update") .sign_with_keys(&Keys::generate()) @@ -501,7 +575,7 @@ mod tests { let updated = build_updated_repo_announcement( &existing, - ProtectionChange::Remove("refs/heads/main".into()), + RepoChange::RemoveProtection("refs/heads/main".into()), ) .expect("build removal") .sign_with_keys(&Keys::generate()) @@ -538,7 +612,7 @@ mod tests { let error = build_updated_repo_announcement( &existing, - ProtectionChange::Set(Box::new(replacement)), + RepoChange::SetProtection(Box::new(replacement)), ) .expect_err("malformed existing rule must fail closed"); @@ -564,7 +638,7 @@ mod tests { let error = build_updated_repo_announcement( &existing, - ProtectionChange::Set(Box::new(replacement)), + RepoChange::SetProtection(Box::new(replacement)), ) .expect_err("the 51st rule must be rejected"); @@ -615,6 +689,145 @@ mod tests { .is_some_and(|error| error.contains("needs pattern + at least one rule"))); } + #[test] + fn bind_channel_replaces_duplicates_and_preserves_everything_else() { + let channel = uuid::Uuid::new_v4().to_string(); + let existing = signed_repo( + vec![ + tag(&["d", "demo"]), + tag(&["name", "Demo"]), + // Two stale bindings — e.g. from a buggy or vanilla client. + tag(&["buzz-channel", "old-and-broken"]), + tag(&["buzz-channel", &uuid::Uuid::new_v4().to_string()]), + tag(&["auth", &"a".repeat(64), "kind=30617", &"b".repeat(128)]), + tag(&["buzz-protect", "refs/heads/main", "push:admin"]), + tag(&["future-metadata", "preserve-me"]), + ], + "repository content", + 100, + ); + + let updated = + build_updated_repo_announcement(&existing, RepoChange::BindChannel(channel.clone())) + .expect("build bind update") + .sign_with_keys(&Keys::generate()) + .expect("sign bind update"); + + assert_eq!(updated.content, "repository content"); + assert_eq!(updated.created_at.as_secs(), 101); + // Exactly one binding remains, and it is the requested one. + let bindings: Vec<_> = updated + .tags + .iter() + .filter(|tag| tag.as_slice().first().map(String::as_str) == Some("buzz-channel")) + .collect(); + assert_eq!(bindings.len(), 1); + assert_eq!(bindings[0].as_slice(), ["buzz-channel", channel.as_str()]); + // Auth stripped (relay re-stamps); everything else preserved. + assert!(!updated + .tags + .iter() + .any(|tag| tag.as_slice().first().map(String::as_str) == Some("auth"))); + assert!(updated + .tags + .iter() + .any(|tag| tag.as_slice() == ["buzz-protect", "refs/heads/main", "push:admin"])); + assert!(updated + .tags + .iter() + .any(|tag| tag.as_slice() == ["future-metadata", "preserve-me"])); + assert!(updated + .tags + .iter() + .any(|tag| tag.as_slice() == ["name", "Demo"])); + } + + #[test] + fn bind_channel_adds_binding_to_unbound_repo() { + let channel = uuid::Uuid::new_v4().to_string(); + let existing = signed_repo(vec![tag(&["d", "demo"])], "", 10); + + let updated = + build_updated_repo_announcement(&existing, RepoChange::BindChannel(channel.clone())) + .expect("build bind update") + .sign_with_keys(&Keys::generate()) + .expect("sign bind update"); + + assert!(updated + .tags + .iter() + .any(|tag| tag.as_slice() == ["buzz-channel", channel.as_str()])); + } + + #[test] + fn bind_channel_rejects_malformed_uuid() { + let existing = signed_repo(vec![tag(&["d", "demo"])], "", 10); + + let error = + build_updated_repo_announcement(&existing, RepoChange::BindChannel("nope".into())) + .expect_err("malformed channel id must not build an update"); + + assert!(matches!(error, crate::error::CliError::Usage(_))); + } + + /// Issue #3527: `repos create --channel` must emit exactly one + /// `buzz-channel` tag so the primary create command stops producing + /// repos the relay 404s forever. + #[test] + fn create_with_channel_emits_exactly_one_binding_tag() { + let channel = uuid::Uuid::new_v4().to_string(); + let event = build_create_announcement( + "demo", + Some("Demo"), + None, + &["https://relay.example/git/owner/demo".to_string()], + None, + &[], + Some(&channel), + ) + .expect("build create announcement") + .sign_with_keys(&Keys::generate()) + .expect("sign create announcement"); + + assert_eq!(event.kind, Kind::Custom(30617)); + let bindings: Vec<_> = event + .tags + .iter() + .filter(|tag| tag.as_slice().first().map(String::as_str) == Some("buzz-channel")) + .collect(); + assert_eq!(bindings.len(), 1, "exactly one buzz-channel tag"); + assert_eq!(bindings[0].as_slice(), ["buzz-channel", channel.as_str()]); + // The standard metadata still rides along. + assert!(event.tags.iter().any(|tag| tag.as_slice() == ["d", "demo"])); + assert!(event + .tags + .iter() + .any(|tag| tag.as_slice() == ["name", "Demo"])); + } + + #[test] + fn create_without_channel_emits_no_binding_tag() { + let event = build_create_announcement("demo", None, None, &[], None, &[], None) + .expect("build create announcement") + .sign_with_keys(&Keys::generate()) + .expect("sign create announcement"); + + assert!( + !event + .tags + .iter() + .any(|tag| tag.as_slice().first().map(String::as_str) == Some("buzz-channel")), + "no --channel means no binding tag (vanilla NIP-34 stays possible)" + ); + } + + #[test] + fn create_rejects_malformed_channel_uuid() { + let error = build_create_announcement("demo", None, None, &[], None, &[], Some("nope")) + .expect_err("malformed channel id must not build an announcement"); + assert!(matches!(error, crate::error::CliError::Usage(_))); + } + #[test] fn duplicate_write_response_is_a_conflict() { let error = validate_write_response( diff --git a/crates/buzz-cli/src/lib.rs b/crates/buzz-cli/src/lib.rs index df02c65be9..02a58a618e 100644 --- a/crates/buzz-cli/src/lib.rs +++ b/crates/buzz-cli/src/lib.rs @@ -1129,6 +1129,11 @@ pub enum ReposCmd { /// Preferred Nostr relay(s) for repo discovery — can be specified multiple times #[arg(long = "nostr-relay")] relays: Vec, + /// Channel UUID to bind the repo to. The `buzz-channel` tag is the + /// git ACL: without it the relay 404s every clone/fetch/push until + /// the author runs `buzz repos bind` (issue #3527). + #[arg(long)] + channel: Option, }, /// Get a repository announcement Get { @@ -1148,6 +1153,20 @@ pub enum ReposCmd { #[arg(long)] limit: Option, }, + /// Bind (or rebind) one of your repositories to a channel. + /// + /// The `buzz-channel` tag on the announcement is the git ACL: the relay + /// authorizes clone/fetch/push by membership in the bound channel. A + /// repo announced without it (e.g. by a vanilla NIP-34 client) returns + /// 404 for everyone until its author binds it here. + Bind { + /// Repository identifier (d-tag). + #[arg(long)] + id: String, + /// Channel UUID to bind. Replaces any existing binding. + #[arg(long)] + channel: String, + }, /// Manage branch and tag protection rules on one of your repositories. #[command(subcommand)] Protect(ReposProtectCmd), @@ -1991,7 +2010,7 @@ mod tests { ); assert_eq!( names(&cmd, "repos"), - vec!["create", "get", "list", "protect"] + vec!["bind", "create", "get", "list", "protect"] ); let repos = cmd .get_subcommands() @@ -2054,7 +2073,7 @@ mod tests { ("patches", 4), ("pr", 5), ("reactions", 3), - ("repos", 4), + ("repos", 5), ("social", 7), ("upload", 1), ("users", 5), diff --git a/crates/buzz-core/src/git_perms.rs b/crates/buzz-core/src/git_perms.rs index 53acd704b9..391781163b 100644 --- a/crates/buzz-core/src/git_perms.rs +++ b/crates/buzz-core/src/git_perms.rs @@ -15,6 +15,30 @@ use crate::channel::MemberRole; use std::fmt; +/// Machine-readable token prefixing the push-policy denial for a kind:30617 +/// announcement with no `buzz-channel` binding. +/// +/// This is a **declared cross-component contract**, not a log string. Known +/// consumers switch on it: +/// - relay `api/git/policy.rs` — produces [`GIT_NO_CHANNEL_BINDING_BODY`] +/// - desktop `src-tauri/commands/project_git_workflow.rs` — merge-failure +/// classifier maps it to a structured `no_channel_binding` error code +/// - desktop `src/features/projects/lib/projectBranchErrors.ts` — dialog +/// copy matcher (TS re-types the literal; its test pins the value) +pub const GIT_NO_CHANNEL_BINDING_TOKEN: &str = "no_channel_binding"; + +/// Full push-policy denial body for an unbound repository. +/// +/// Format: `: `. The trailing prose deliberately +/// repeats the token's meaning because desktops already in the field match +/// the exact phrase `no channel binding` (spaces, not underscores — the +/// token alone would NOT satisfy that matcher). Do not "fix" the redundancy: +/// removing the phrase silently breaks every shipped desktop, and removing +/// the token breaks the structured consumers above. A relay-side test pins +/// both matchers. +pub const GIT_NO_CHANNEL_BINDING_BODY: &str = + "no_channel_binding: repository has no channel binding"; + /// Maximum number of `buzz-protect` tags per repo. pub const MAX_PROTECTION_RULES: usize = 50; /// Maximum character length of a ref pattern. diff --git a/crates/buzz-relay/src/api/git/binding.rs b/crates/buzz-relay/src/api/git/binding.rs new file mode 100644 index 0000000000..7ee0eccb23 --- /dev/null +++ b/crates/buzz-relay/src/api/git/binding.rs @@ -0,0 +1,128 @@ +//! Repo → channel binding resolution, shared by the read gate and push policy. +//! +//! The `buzz-channel` tag on a kind:30617 announcement IS the git ACL: the +//! read gate (SEC-005, `transport::authorize_git_read`) and the push policy +//! endpoint (`policy::hook_callback`) both authorize against membership in +//! the bound channel. Before this module they each parsed the tag with their +//! own code that agreed only by coincidence; the resolver makes the +//! agreement structural. +//! +//! # First-tag, fail-closed semantics +//! +//! Only the *first* `buzz-channel` tag is considered, and it must carry a +//! valid UUID. A malformed first binding resolves to [`RepoBinding::Broken`] +//! even if a later duplicate tag is valid — an ambiguous announcement must +//! fail closed, not silently resolve to whichever duplicate happens to +//! parse. If this ever became "find the first *parseable* tag", an author +//! who can append a second `buzz-channel` tag would pick the channel. +//! +//! # What this deliberately does NOT do +//! +//! No DB access. A well-formed UUID that names a nonexistent or deleted +//! channel still resolves to [`RepoBinding::Bound`]; each gate's own +//! membership lookup then denies (`get_member_role` joins +//! `channels … deleted_at IS NULL`, so a dead channel is indistinguishable +//! from a non-member — the info-leak-safe posture). Likewise each gate keeps +//! its own archived-channel policy: push denies on archived channels, read +//! does not, and this resolver must not unify that asymmetry as a side +//! effect. + +use uuid::Uuid; + +/// How a kind:30617 announcement binds (or fails to bind) a channel. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RepoBinding { + /// No `buzz-channel` tag at all. The announcement author (the only + /// identity that can rebind — 30617 is keyed by `(author, d)`) may be + /// offered remediation; everyone else gets the generic denial. + NotBound, + /// First `buzz-channel` tag carries a valid UUID. + Bound(Uuid), + /// First `buzz-channel` tag exists but its value is not a UUID. + /// Fail closed with the generic denial — never remediation, which + /// would leak that the repo exists. + Broken, +} + +/// Resolve the channel binding of a kind:30617 announcement from its tags. +pub fn resolve_repo_binding(event: &nostr::Event) -> RepoBinding { + let Some(first) = event + .tags + .iter() + .find(|t| t.as_slice().first().map(String::as_str) == Some("buzz-channel")) + else { + return RepoBinding::NotBound; + }; + match first.as_slice().get(1).map(|v| Uuid::parse_str(v)) { + Some(Ok(id)) => RepoBinding::Bound(id), + _ => RepoBinding::Broken, + } +} + +#[cfg(test)] +mod tests { + use nostr::{EventBuilder, Keys, Kind, Tag}; + + use super::{resolve_repo_binding, RepoBinding}; + + fn announcement(tags: Vec) -> nostr::Event { + EventBuilder::new(Kind::Custom(30617), "") + .tags(tags) + .sign_with_keys(&Keys::generate()) + .expect("sign 30617") + } + + #[test] + fn extracts_valid_uuid() { + let ch = uuid::Uuid::new_v4(); + let event = announcement(vec![ + Tag::parse(["d", "repo"]).unwrap(), + Tag::parse(["buzz-channel", &ch.to_string()]).unwrap(), + ]); + assert_eq!(resolve_repo_binding(&event), RepoBinding::Bound(ch)); + } + + #[test] + fn absent_tag_is_not_bound() { + let event = announcement(vec![Tag::parse(["d", "repo"]).unwrap()]); + assert_eq!(resolve_repo_binding(&event), RepoBinding::NotBound); + } + + #[test] + fn malformed_and_empty_values_are_broken_not_absent() { + let malformed = announcement(vec![ + Tag::parse(["d", "repo"]).unwrap(), + Tag::parse(["buzz-channel", "not-a-uuid"]).unwrap(), + ]); + assert_eq!(resolve_repo_binding(&malformed), RepoBinding::Broken); + + let empty = announcement(vec![ + Tag::parse(["d", "repo"]).unwrap(), + Tag::parse(["buzz-channel"]).unwrap(), + ]); + assert_eq!(resolve_repo_binding(&empty), RepoBinding::Broken); + } + + #[test] + fn fails_closed_on_ambiguous_duplicate_bindings() { + let ch = uuid::Uuid::new_v4(); + let other = uuid::Uuid::new_v4(); + + // Malformed first + valid second: the ambiguity denies; the valid + // duplicate must NOT win, or the duplicate picks the channel. + let malformed_first = announcement(vec![ + Tag::parse(["d", "repo"]).unwrap(), + Tag::parse(["buzz-channel", "not-a-uuid"]).unwrap(), + Tag::parse(["buzz-channel", &ch.to_string()]).unwrap(), + ]); + assert_eq!(resolve_repo_binding(&malformed_first), RepoBinding::Broken); + + // Valid first + different second: first wins deterministically. + let valid_first = announcement(vec![ + Tag::parse(["d", "repo"]).unwrap(), + Tag::parse(["buzz-channel", &ch.to_string()]).unwrap(), + Tag::parse(["buzz-channel", &other.to_string()]).unwrap(), + ]); + assert_eq!(resolve_repo_binding(&valid_first), RepoBinding::Bound(ch)); + } +} diff --git a/crates/buzz-relay/src/api/git/mod.rs b/crates/buzz-relay/src/api/git/mod.rs index ab0510fbeb..dd69d7dc36 100644 --- a/crates/buzz-relay/src/api/git/mod.rs +++ b/crates/buzz-relay/src/api/git/mod.rs @@ -22,6 +22,7 @@ use tower_http::limit::RequestBodyLimitLayer; use crate::state::AppState; +pub mod binding; pub mod cas_publish; pub mod hook; pub mod hydrate; diff --git a/crates/buzz-relay/src/api/git/policy.rs b/crates/buzz-relay/src/api/git/policy.rs index fd6c4fb688..32d63f4600 100644 --- a/crates/buzz-relay/src/api/git/policy.rs +++ b/crates/buzz-relay/src/api/git/policy.rs @@ -42,7 +42,10 @@ use tracing::{error, warn}; use uuid::Uuid; use buzz_core::channel::MemberRole; -use buzz_core::git_perms::{evaluate_push, parse_protection_tags, Denial, RefUpdate, UpdateKind}; +use buzz_core::git_perms::{ + evaluate_push, parse_protection_tags, Denial, RefUpdate, UpdateKind, + GIT_NO_CHANNEL_BINDING_BODY, +}; use buzz_db::EventQuery; use crate::state::AppState; @@ -297,12 +300,29 @@ pub async fn hook_policy_check( } }; - // 6. Resolve channel and check archived state (applies to ALL pushers including owner). - let channel_id = tags - .iter() - .find(|t| t.first().map(|s| s.as_str()) == Some("buzz-channel")) - .and_then(|t| t.get(1)) - .and_then(|id| Uuid::parse_str(id).ok()); + // 6. Resolve channel binding via the shared resolver (same first-tag, + // fail-closed semantics as the read gate) and check archived state + // (applies to ALL pushers including owner). + // + // `Broken` denies HERE, before owner resolution: a malformed or + // ambiguous first binding fails closed for *everyone*, exactly like the + // read gate. Letting it fall through as "unbound" would hand the owner + // short-circuit below a push path through a binding the read gate + // refuses to honor — the tri-state exists precisely so Broken and + // NotBound cannot collapse. Only genuinely-NotBound repos proceed, and + // only they may earn the remediation-token denial. + let channel_id = match crate::api::git::binding::resolve_repo_binding(&repo_event.event) { + crate::api::git::binding::RepoBinding::Bound(id) => Some(id), + crate::api::git::binding::RepoBinding::NotBound => None, + crate::api::git::binding::RepoBinding::Broken => { + warn!(repo = %req.repo_id, "hook callback: broken buzz-channel binding"); + // Deliberately NOT the no_channel_binding token body: the + // remediation contract is NotBound-only. A broken binding is + // ambiguity, and ambiguity gets a generic denial (matching the + // read gate's posture for the same announcement). + return (StatusCode::FORBIDDEN, "invalid channel binding").into_response(); + } + }; if let Some(ch_id) = channel_id { match state.db.get_channel(community, ch_id).await { @@ -350,7 +370,10 @@ pub async fn hook_policy_check( match channel_id { None => { warn!(repo = %req.repo_id, "hook callback: no buzz-channel binding"); - return (StatusCode::FORBIDDEN, "no channel binding").into_response(); + // Declared cross-component contract — see the const docs in + // buzz-core::git_perms for who consumes the token and why + // the body also repeats the legacy phrase. + return (StatusCode::FORBIDDEN, GIT_NO_CHANNEL_BINDING_BODY).into_response(); } Some(ch_id) => { match state @@ -482,6 +505,30 @@ mod tests { assert!(!verify_hmac(b"wrong-secret", &req)); } + /// Deploy-skew guard for the unbound-repo deny body. The token + /// (`no_channel_binding`, underscores) and the legacy phrase + /// (`no channel binding`, spaces) do NOT contain each other, so the body + /// must carry both: the token for structured consumers (Desktop's merge + /// classifier and dialog matcher), the phrase for desktops already in + /// the field that prose-match it. Relay ships continuously and Desktop + /// on release cadence — dropping the phrase strands every old desktop + /// on a new relay. Asserted against the shared consts, not re-typed + /// literals, so the const and this test cannot drift apart separately. + #[test] + fn no_channel_binding_body_satisfies_old_and_new_matchers() { + assert!( + GIT_NO_CHANNEL_BINDING_BODY.starts_with(&format!( + "{}: ", + buzz_core::git_perms::GIT_NO_CHANNEL_BINDING_TOKEN + )), + "new structured consumers match the token prefix" + ); + assert!( + GIT_NO_CHANNEL_BINDING_BODY.contains("no channel binding"), + "shipped desktops prose-match this exact phrase (spaces, not underscores)" + ); + } + #[test] fn hmac_tampered_repo_id_rejected() { let secret = b"test-secret"; @@ -772,4 +819,169 @@ printf '%s' "$HMAC_INPUT" | openssl dgst -sha256 -hmac "{secret}" -hex 2>/dev/nu "Single-ref HMAC mismatch!\n Rust: {rust_sig}\n Bash: {bash_sig}" ); } + + // ── hook_policy_check binding gate (requires Postgres) ────────────── + + const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz"; // sadscan:disable np.postgres.1 + + async fn policy_test_state() -> Arc { + let mut config = crate::config::Config::from_env().expect("default config loads"); + config.require_relay_membership = false; + config.redis_url = "redis://127.0.0.1:1".to_string(); + config.database_url = std::env::var("BUZZ_TEST_DATABASE_URL") + .or_else(|_| std::env::var("DATABASE_URL")) + .unwrap_or_else(|_| TEST_DB_URL.to_string()); + let pool = sqlx::PgPool::connect(&config.database_url) + .await + .expect("connect test DB"); + let db = buzz_db::Db::from_pool(pool.clone()); + let redis_pool = deadpool_redis::Config::from_url(&config.redis_url) + .create_pool(Some(deadpool_redis::Runtime::Tokio1)) + .expect("redis pool"); + let pubsub = Arc::new( + buzz_pubsub::PubSubManager::new(&config.redis_url, redis_pool.clone()) + .await + .expect("pubsub manager"), + ); + let audit = buzz_audit::AuditService::new(pool.clone()); + let auth = buzz_auth::AuthService::new(config.auth.clone()); + let search = buzz_search::SearchService::new(pool.clone()); + let workflow_engine = Arc::new(buzz_workflow::WorkflowEngine::new( + db.clone(), + buzz_workflow::WorkflowConfig::default(), + )); + let media_storage = buzz_media::MediaStorage::new(&config.media).expect("media storage"); + let (state, _audit_shutdown) = AppState::new( + config, + db, + redis_pool, + audit, + pubsub, + auth, + search, + workflow_engine, + nostr::Keys::generate(), + media_storage, + ); + Arc::new(state) + } + + /// Announce `repo_id` with the given tags, then push to it as its own + /// announcement author and return the response. + async fn owner_push_response( + state: &Arc, + community: buzz_core::CommunityId, + keys: &nostr::Keys, + repo_id: &str, + binding_tags: Vec, + ) -> axum::response::Response { + use nostr::{EventBuilder, Kind, Tag}; + + let mut tags = vec![Tag::parse(["d", repo_id]).unwrap()]; + tags.extend(binding_tags); + let event = EventBuilder::new(Kind::Custom(30617), "") + .tags(tags) + .sign_with_keys(keys) + .expect("sign 30617"); + state + .db + .insert_event(community, &event, None) + .await + .expect("insert 30617"); + + let owner_hex = keys.public_key().to_hex(); + let mut req = HookCallbackRequest { + repo_id: repo_id.to_string(), + repo_owner: owner_hex.clone(), + community_id: community.as_uuid().to_string(), + pusher_pubkey: owner_hex, + ref_updates: vec![HookRefUpdate { + old_oid: "0".repeat(40), + new_oid: "2".repeat(40), + ref_name: "refs/heads/main".to_string(), + is_ancestor: false, + }], + timestamp: SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_secs(), + signature: String::new(), + }; + let secret = state.config.git_hook_hmac_secret.clone(); + sign_request(&mut req, secret.as_bytes()); + hook_policy_check(State(Arc::clone(state)), Json(req)).await + } + + async fn body_string(response: axum::response::Response) -> (StatusCode, String) { + let status = response.status(); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("read body"); + (status, String::from_utf8(bytes.to_vec()).expect("utf-8")) + } + + /// The tri-state trap the resolver exists to prevent: a broken (malformed + /// or ambiguous-first) binding must fail closed for EVERYONE on push — + /// including the announcement author — *before* the owner short-circuit + /// grants `MemberRole::Owner`. Collapsing `Broken` into "unbound" hands + /// the owner a push path through a binding the read gate refuses to + /// honor. The remediation token stays reserved for genuinely NotBound. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn push_gate_denies_owner_through_broken_binding() { + use nostr::{Keys, Tag}; + + let state = policy_test_state().await; + let host = format!("policy-{}.example", uuid::Uuid::new_v4().simple()); + let community = state + .db + .ensure_configured_community(&host) + .await + .expect("community") + .id; + let keys = Keys::generate(); + + // Malformed first + valid-looking second: the ambiguity must deny, + // and the parseable duplicate must not rescue the push. + let response = owner_push_response( + &state, + community, + &keys, + &format!("repo-{}", uuid::Uuid::new_v4().simple()), + vec![ + Tag::parse(["buzz-channel", "not-a-uuid"]).unwrap(), + Tag::parse(["buzz-channel", &uuid::Uuid::new_v4().to_string()]).unwrap(), + ], + ) + .await; + let (status, body) = body_string(response).await; + assert_eq!(status, StatusCode::FORBIDDEN); + assert_eq!( + body, "invalid channel binding", + "owner pushing through a broken binding must be denied generically" + ); + assert!( + !body.contains(buzz_core::git_perms::GIT_NO_CHANNEL_BINDING_TOKEN), + "remediation token is NotBound-only; Broken must never earn it" + ); + + // Control: the same owner pushing a genuinely NEVER-BOUND repo is + // allowed (owner authority over an unbound announcement is the + // long-standing push semantics). This pins the denial above to + // Broken specifically, not to some broader regression. + let response = owner_push_response( + &state, + community, + &keys, + &format!("repo-{}", uuid::Uuid::new_v4().simple()), + vec![], + ) + .await; + let (status, body) = body_string(response).await; + assert_eq!( + status, + StatusCode::OK, + "owner push to a never-bound repo must remain allowed (got body: {body})" + ); + } } diff --git a/crates/buzz-relay/src/api/git/transport.rs b/crates/buzz-relay/src/api/git/transport.rs index df5bdd4c3e..11c4f6d35b 100644 --- a/crates/buzz-relay/src/api/git/transport.rs +++ b/crates/buzz-relay/src/api/git/transport.rs @@ -28,6 +28,7 @@ use tokio::process::Command; use tower_http::limit::RequestBodyLimitLayer; use tracing::{error, info, warn}; +use super::binding::{resolve_repo_binding, RepoBinding}; use super::cas_publish::{cas_publish, CasError, ParentState, PublishLimits}; use super::hook::install_hook; use super::hydrate::{ @@ -377,7 +378,15 @@ fn hydrate_error_to_response(owner: &str, repo: &str, err: HydrateError) -> Resp /// error all deny. There is deliberately **no repo-owner bypass**: an owner /// removed from the bound channel loses read access, which is the exact /// exploit shape this gate closes. Every denial is the same generic 404 as a -/// nonexistent repo so membership cannot be probed through the git endpoints. +/// nonexistent repo so membership cannot be probed through the git endpoints +/// — with exactly one carve-out: a **never-bound** repo read by its own +/// **announcement author** returns a 404 whose body tells the author how to +/// bind it (issue #3527: a vanilla NIP-34 client can announce without a +/// `buzz-channel` tag, and the repo then 404s forever with no explanation +/// for anyone). The author already knows the repo exists — they announced it +/// — so the remediation body leaks nothing, and only the author can rebind +/// (kind:30617 is keyed by `(author, d)`). A *broken* binding stays generic +/// even for the author: ambiguity fails closed. async fn authorize_git_read( db: &buzz_db::Db, community: buzz_core::CommunityId, @@ -415,9 +424,32 @@ async fn authorize_git_read( } }; - let Some(channel_id) = repo_bound_channel_id(&repo_event.event) else { - warn!(repo = %repo_name, "git read gate: missing/malformed buzz-channel binding (deny)"); - return Err(denied()); + let channel_id = match resolve_repo_binding(&repo_event.event) { + RepoBinding::Bound(id) => id, + RepoBinding::NotBound => { + // Remediation carve-out: author of a never-bound announcement. + // Status stays 404 — byte-identical to every other denial at the + // status level — so denial *class* is still unprobeable; only + // the body differs, and only for the one identity that already + // knows the repo exists. The body is a single verb-first line: + // Desktop error paths that keep one line keep the instruction. + if repo_event.event.pubkey == *caller { + warn!(repo = %repo_name, "git read gate: unbound repo read by its author (deny with remediation)"); + return Err(( + StatusCode::NOT_FOUND, + format!( + "run: buzz repos bind --id {repo_name} --channel — repository {repo_name:?} has no channel binding, so the relay cannot authorize access" + ), + ) + .into_response()); + } + warn!(repo = %repo_name, "git read gate: missing buzz-channel binding (deny)"); + return Err(denied()); + } + RepoBinding::Broken => { + warn!(repo = %repo_name, "git read gate: malformed buzz-channel binding (deny)"); + return Err(denied()); + } }; match db @@ -433,24 +465,6 @@ async fn authorize_git_read( } } -/// Extract the `buzz-channel` UUID from a kind:30617 announcement. -/// -/// First-tag semantics, matching the push policy endpoint: only the *first* -/// `buzz-channel` tag is considered, and it must carry a valid UUID. A -/// malformed first binding denies even if a later duplicate tag is valid — -/// an ambiguous announcement must fail closed, not silently resolve to -/// whichever duplicate happens to parse. -fn repo_bound_channel_id(event: &nostr::Event) -> Option { - let first = event - .tags - .iter() - .find(|t| t.as_slice().first().map(String::as_str) == Some("buzz-channel"))?; - first - .as_slice() - .get(1) - .and_then(|v| uuid::Uuid::parse_str(v).ok()) -} - /// Pure decision for [`authorize_git_read`]: a read requires a current /// active membership row whose role the relay recognizes. /// @@ -2454,75 +2468,30 @@ mod sec005_read_gate_tests { .expect("sign 30617") } - #[test] - fn repo_bound_channel_id_extracts_valid_uuid() { - let keys = Keys::generate(); - let ch = uuid::Uuid::new_v4(); - let event = announcement( - &keys, - vec![ - Tag::parse(["d", "r"]).unwrap(), - Tag::parse(["buzz-channel", &ch.to_string()]).unwrap(), - ], - ); - assert_eq!(repo_bound_channel_id(&event), Some(ch)); - } - - #[test] - fn repo_bound_channel_id_rejects_absent_and_malformed_bindings() { - let keys = Keys::generate(); - let absent = announcement(&keys, vec![Tag::parse(["d", "r"]).unwrap()]); - assert_eq!(repo_bound_channel_id(&absent), None); - - let malformed = announcement( - &keys, - vec![ - Tag::parse(["d", "r"]).unwrap(), - Tag::parse(["buzz-channel", "not-a-uuid"]).unwrap(), - ], - ); - assert_eq!(repo_bound_channel_id(&malformed), None); - - let empty = announcement( - &keys, - vec![ - Tag::parse(["d", "r"]).unwrap(), - Tag::parse(["buzz-channel"]).unwrap(), - ], - ); - assert_eq!(repo_bound_channel_id(&empty), None); + // Binding *parse* semantics (first-tag fails-closed, duplicate-tag + // ambiguity, malformed vs. absent) are unit-tested where the resolver + // lives: `super::super::binding`. The tests below prove the *gate* wires + // each resolver outcome to the right response — allow, generic denial + // body, or the author remediation body — which the resolver tests + // cannot see. + + /// Collapse an `authorize_git_read` denial to `(status, body)` so tests + /// can assert on the exact bytes a git client would see. A blind + /// `.is_err()` cannot distinguish the generic 404 from the remediation + /// 404 — and that distinction IS the security property. + async fn denial_parts(result: Result<(), Response>) -> (StatusCode, String) { + let response = result.expect_err("expected a denial"); + let status = response.status(); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("read denial body"); + ( + status, + String::from_utf8(bytes.to_vec()).expect("utf-8 body"), + ) } - #[test] - fn repo_bound_channel_id_fails_closed_on_ambiguous_duplicate_bindings() { - // First-tag semantics: a malformed first binding must deny even when - // a later duplicate tag is valid. An ambiguous announcement must not - // silently resolve to whichever duplicate happens to parse. - let keys = Keys::generate(); - let ch = uuid::Uuid::new_v4(); - let malformed_first = announcement( - &keys, - vec![ - Tag::parse(["d", "r"]).unwrap(), - Tag::parse(["buzz-channel", "not-a-uuid"]).unwrap(), - Tag::parse(["buzz-channel", &ch.to_string()]).unwrap(), - ], - ); - assert_eq!(repo_bound_channel_id(&malformed_first), None); - - // And the mirror image: a valid first binding wins, matching the - // push policy endpoint's first-tag resolution. - let other = uuid::Uuid::new_v4(); - let valid_first = announcement( - &keys, - vec![ - Tag::parse(["d", "r"]).unwrap(), - Tag::parse(["buzz-channel", &ch.to_string()]).unwrap(), - Tag::parse(["buzz-channel", &other.to_string()]).unwrap(), - ], - ); - assert_eq!(repo_bound_channel_id(&valid_first), Some(ch)); - } + const GENERIC_DENIAL: &str = "repository not found"; // ── authorize_git_read matrix (requires Postgres) ──────────────────── @@ -2544,6 +2513,11 @@ mod sec005_read_gate_tests { Missing, /// `buzz-channel` tag whose value is not a UUID. Malformed, + /// `buzz-channel` tag carrying a well-formed UUID that names no + /// channel. The resolver reports `Bound`; the membership lookup + /// (whose SQL joins `channels … deleted_at IS NULL`) then returns + /// no role — the deliberate phase-1 posture for dead bindings. + UnknownChannel, } struct RepoFixture { @@ -2609,6 +2583,9 @@ mod sec005_read_gate_tests { Binding::Malformed => { tags.push(Tag::parse(["buzz-channel", "not-a-uuid"]).unwrap()); } + Binding::UnknownChannel => { + tags.push(Tag::parse(["buzz-channel", &uuid::Uuid::new_v4().to_string()]).unwrap()); + } } let event = announcement(&owner_keys, tags); db.insert_event(community, &event, None) @@ -2676,32 +2653,63 @@ mod sec005_read_gate_tests { #[tokio::test] #[ignore = "requires Postgres"] async fn read_gate_denies_missing_or_malformed_binding_and_absent_repo() { - // Missing buzz-channel tag → deny even for a channel member. + // Missing buzz-channel tag → deny even for a channel member, with + // the generic body: the remediation carve-out is author-only. let f = setup_repo(Binding::Missing).await; let member = f.member_keys.public_key(); - assert!( - authorize_git_read(&f.db, f.community, &member, &f.owner_hex, &f.repo) - .await - .is_err(), - "announcement without buzz-channel binding must deny" + let (status, body) = denial_parts( + authorize_git_read(&f.db, f.community, &member, &f.owner_hex, &f.repo).await, + ) + .await; + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!( + body, GENERIC_DENIAL, + "unbound repo read by a NON-author must get the generic body — \ + remediation for anyone but the announcement author leaks repo existence" ); - // Malformed buzz-channel tag → deny. + // Malformed buzz-channel tag → deny with the generic body EVEN FOR + // THE AUTHOR. This is the assertion that pins the carve-out to + // NotBound: if it ever fires on Broken, this fails on bytes, not + // on Ok/Err (which cannot see the difference). let g = setup_repo(Binding::Malformed).await; - let member_g = g.member_keys.public_key(); - assert!( - authorize_git_read(&g.db, g.community, &member_g, &g.owner_hex, &g.repo) - .await - .is_err(), - "announcement with malformed buzz-channel binding must deny" + let g_owner = g.owner_keys.public_key(); + let (status, body) = denial_parts( + authorize_git_read(&g.db, g.community, &g_owner, &g.owner_hex, &g.repo).await, + ) + .await; + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!( + body, GENERIC_DENIAL, + "broken binding must stay generic even for the author (ambiguity fails closed)" + ); + + // Well-formed UUID naming a nonexistent channel → resolver says + // Bound, membership lookup finds nothing → generic denial for + // everyone, author included. The dead-channel case must be + // indistinguishable from non-membership (phase-1 posture; ingest + // validation closes the front door in phase 2). + let u = setup_repo(Binding::UnknownChannel).await; + let u_owner = u.owner_keys.public_key(); + let (status, body) = denial_parts( + authorize_git_read(&u.db, u.community, &u_owner, &u.owner_hex, &u.repo).await, + ) + .await; + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!( + body, GENERIC_DENIAL, + "binding to a nonexistent channel must deny generically, even for the author" ); // Nonexistent announcement → deny. - assert!( - authorize_git_read(&f.db, f.community, &member, &f.owner_hex, "no-such-repo") - .await - .is_err(), - "nonexistent repo must deny" + let (status, body) = denial_parts( + authorize_git_read(&f.db, f.community, &member, &f.owner_hex, "no-such-repo").await, + ) + .await; + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!( + body, GENERIC_DENIAL, + "nonexistent repo must deny generically" ); // Owner-mismatch: URL owner differs from announcement author → deny. @@ -2722,6 +2730,53 @@ mod sec005_read_gate_tests { ); } + #[tokio::test] + #[ignore = "requires Postgres"] + async fn read_gate_gives_author_of_unbound_repo_remediation_body() { + // Issue #3527: the author of a never-bound announcement is the one + // identity that can fix it (30617 is keyed by (author, d)) and the + // one identity remediation cannot leak anything to. Status must stay + // 404 — identical to every other denial — with the bind command in + // the body. + let f = setup_repo(Binding::Missing).await; + let author = f.owner_keys.public_key(); + + let response = authorize_git_read(&f.db, f.community, &author, &f.owner_hex, &f.repo) + .await + .expect_err("unbound repo must still deny its author"); + assert_eq!(response.status(), StatusCode::NOT_FOUND); + // Guard against a future "tidy" into Json(...) or a custom + // IntoResponse: git prints `remote:` lines only for text/plain + // bodies — any other content-type makes the remediation silently + // invisible in the user's terminal with no failing assertion. + assert_eq!( + response + .headers() + .get(axum::http::header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()), + Some("text/plain; charset=utf-8"), + "remediation body must stay text/plain or git clients will swallow it" + ); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("read remediation body"); + let body = String::from_utf8(bytes.to_vec()).expect("utf-8 body"); + assert!( + body.starts_with(&format!("run: buzz repos bind --id {}", f.repo)), + "remediation must lead with the actionable command (got {body:?})" + ); + assert_ne!(body, GENERIC_DENIAL); + + // Same repo, same state, different caller: a member of some channel + // who is not the author still gets the generic body. + let member = f.member_keys.public_key(); + let (_, body) = denial_parts( + authorize_git_read(&f.db, f.community, &member, &f.owner_hex, &f.repo).await, + ) + .await; + assert_eq!(body, GENERIC_DENIAL, "remediation is author-only"); + } + #[tokio::test] #[ignore = "requires Postgres"] async fn read_gate_follows_current_announcement_not_stale_registry() { diff --git a/crates/buzz-relay/src/handlers/ingest.rs b/crates/buzz-relay/src/handlers/ingest.rs index a30b0e714d..ee644d5a9b 100644 --- a/crates/buzz-relay/src/handlers/ingest.rs +++ b/crates/buzz-relay/src/handlers/ingest.rs @@ -2481,7 +2481,12 @@ async fn ingest_event_inner( crate::handlers::side_effects::handle_side_effects(tenant, kind_u32, &event, state) .await { - warn!(event_id = %event_id_hex, kind = kind_u32, "Side effect failed: {e}"); + // error!, not warn!: the event was accepted but its side effects + // (channel creation, git repo seeding, …) did not run — the relay + // is now in a state the client believes it isn't. Production runs + // RUST_LOG=error, so warn! made these failures invisible during + // the #3527 triage. + error!(event_id = %event_id_hex, kind = kind_u32, "Side effect failed: {e}"); } } diff --git a/crates/buzz-test-client/tests/e2e_git.rs b/crates/buzz-test-client/tests/e2e_git.rs index f543e56229..3c82e31764 100644 --- a/crates/buzz-test-client/tests/e2e_git.rs +++ b/crates/buzz-test-client/tests/e2e_git.rs @@ -62,6 +62,27 @@ async fn post_event(event: &nostr::Event) { ); } +/// Create a channel (kind:9007) owned by `keys` and return its UUID. +/// +/// The git read gate (SEC-005) authorizes against membership in the channel +/// named by the announcement's `buzz-channel` tag, so every repo these tests +/// announce must be bound to a channel its owner belongs to — creating the +/// channel makes the creator its owner-member. +async fn create_test_channel(keys: &Keys) -> String { + let channel_uuid = uuid::Uuid::new_v4().to_string(); + let event = EventBuilder::new(Kind::Custom(9007), "") + .tags(vec![ + Tag::parse(["h", &channel_uuid]).unwrap(), + Tag::parse(["name", &format!("git-e2e-{channel_uuid}")]).unwrap(), + Tag::parse(["channel_type", "stream"]).unwrap(), + Tag::parse(["visibility", "open"]).unwrap(), + ]) + .sign_with_keys(keys) + .unwrap(); + post_event(&event).await; + channel_uuid +} + /// Run `git` with the Buzz credential helper and isolated config. fn git_status(args: &[&str], cwd: &Path, owner_nsec: &str) -> std::process::Output { let helper = credential_helper(); @@ -256,10 +277,15 @@ async fn git_clone_push_fetch_force_roundtrip() { let s3 = GitS3Probe::from_env(); // Announce the repo (kind:30617) so the relay creates the bare repo + hook. + // The `buzz-channel` binding is the repo's ACL: without it the read gate + // 404s even for the owner (issue #3527), so bind to a channel the owner + // just created (and therefore belongs to). + let channel = create_test_channel(&owner).await; let announce = EventBuilder::new(Kind::from(30617), "") .tags(vec![ Tag::parse(["d", &repo]).unwrap(), Tag::parse(["name", "e2e git repo"]).unwrap(), + Tag::parse(["buzz-channel", &channel]).unwrap(), ]) .sign_with_keys(&owner) .unwrap(); @@ -393,10 +419,12 @@ async fn git_concurrent_push_one_wins_and_repo_recovers() { let repo = format!("e2e-git-concurrent-{}", std::process::id()); let s3 = GitS3Probe::from_env(); + let channel = create_test_channel(&owner).await; let announce = EventBuilder::new(Kind::from(30617), "") .tags(vec![ Tag::parse(["d", &repo]).unwrap(), Tag::parse(["name", "e2e concurrent git repo"]).unwrap(), + Tag::parse(["buzz-channel", &channel]).unwrap(), ]) .sign_with_keys(&owner) .unwrap(); diff --git a/desktop/src-tauri/src/commands/mod.rs b/desktop/src-tauri/src/commands/mod.rs index 1c89ee4f77..66ef7ef17b 100644 --- a/desktop/src-tauri/src/commands/mod.rs +++ b/desktop/src-tauri/src/commands/mod.rs @@ -44,6 +44,7 @@ mod project_git; mod project_git_branches; mod project_git_diff; mod project_git_exec; +mod project_git_merge_error; mod project_git_push; mod project_git_workflow; mod project_repo_paths; diff --git a/desktop/src-tauri/src/commands/project_git_merge_error.rs b/desktop/src-tauri/src/commands/project_git_merge_error.rs new file mode 100644 index 0000000000..460ce83fa3 --- /dev/null +++ b/desktop/src-tauri/src/commands/project_git_merge_error.rs @@ -0,0 +1,152 @@ +//! Structured pull-request merge failures returned across the Tauri boundary. + +use serde::Serialize; + +/// Machine-readable recovery metadata for a failed pull-request merge. +#[derive(Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ProjectPullRequestMergeRecovery { + action: String, + target_branch: String, + source_branch: String, +} + +/// Structured pull-request merge failure returned across the Tauri boundary. +#[derive(Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ProjectPullRequestMergeError { + code: String, + message: String, + recovery: Option, +} + +impl ProjectPullRequestMergeError { + pub(crate) fn new(code: &str, message: impl Into) -> Self { + Self { + code: code.to_string(), + message: message.into(), + recovery: None, + } + } + + fn conflict(target_branch: String, source_branch: String) -> Self { + Self { + code: "merge_conflict".to_string(), + message: "Pull request has merge conflicts.".to_string(), + recovery: Some(ProjectPullRequestMergeRecovery { + action: "open_terminal".to_string(), + target_branch, + source_branch, + }), + } + } +} + +impl From for ProjectPullRequestMergeError { + fn from(message: String) -> Self { + // Relay push-policy denial for a repo with no `buzz-channel` binding. + // The stable token is declared in `buzz-core::git_perms` + // (GIT_NO_CHANNEL_BINDING_TOKEN); the relay guarantees the denial body + // starts with it. Push failures reach this conversion as raw + // stderr/`remote:` text, so match the token anywhere in the message. + if message.contains(buzz_core_pkg::git_perms::GIT_NO_CHANNEL_BINDING_TOKEN) { + return Self::new( + buzz_core_pkg::git_perms::GIT_NO_CHANNEL_BINDING_TOKEN, + "This repository is not bound to a channel, so the relay cannot \ + authorize pushes. Bind it with: buzz repos bind --id \ + --channel ", + ); + } + Self::new("merge_failed", message) + } +} + +pub(crate) fn classify_merge_error( + message: String, + has_conflicts: bool, + target_branch: &str, + source_branch: &str, +) -> ProjectPullRequestMergeError { + if has_conflicts { + ProjectPullRequestMergeError::conflict(target_branch.to_string(), source_branch.to_string()) + } else { + ProjectPullRequestMergeError::new( + "merge_failed", + format!("Pull request merge failed: {message}"), + ) + } +} + +#[cfg(test)] +mod tests { + use super::{classify_merge_error, ProjectPullRequestMergeError}; + + #[test] + fn merge_conflict_error_has_stable_recovery_metadata() { + let error = + ProjectPullRequestMergeError::conflict("main".to_string(), "feature/demo".to_string()); + + assert_eq!(error.code, "merge_conflict"); + assert_eq!(error.message, "Pull request has merge conflicts."); + let recovery = error.recovery.expect("conflict recovery"); + assert_eq!(recovery.action, "open_terminal"); + assert_eq!(recovery.target_branch, "main"); + assert_eq!(recovery.source_branch, "feature/demo"); + } + + #[test] + fn merge_conflict_error_serializes_for_tauri_clients() { + let error = + ProjectPullRequestMergeError::conflict("main".to_string(), "feature/demo".to_string()); + let value = serde_json::to_value(error).expect("serialize merge conflict"); + + assert_eq!(value["code"], "merge_conflict"); + assert_eq!(value["recovery"]["targetBranch"], "main"); + assert_eq!(value["recovery"]["sourceBranch"], "feature/demo"); + } + + #[test] + fn merge_error_classification_only_recovers_conflicts() { + let conflict = classify_merge_error( + "CONFLICT (content): Merge conflict in src/main.rs".to_string(), + true, + "main", + "feature/demo", + ); + assert_eq!(conflict.code, "merge_conflict"); + assert!(conflict.recovery.is_some()); + + let other = classify_merge_error( + "fatal: refusing to merge unrelated histories".to_string(), + false, + "main", + "feature/demo", + ); + assert_eq!(other.code, "merge_failed"); + assert!(other.recovery.is_none()); + } + + #[test] + fn no_channel_binding_denial_converts_to_structured_code() { + // The relay's push-policy denial arrives as raw git stderr with + // `remote:` framing; the stable token must be recognized wherever it + // sits in the message. + let remote_stderr = format!( + "remote: {}\nerror: failed to push some refs", + buzz_core_pkg::git_perms::GIT_NO_CHANNEL_BINDING_BODY + ); + let error = ProjectPullRequestMergeError::from(remote_stderr); + + assert_eq!( + error.code, + buzz_core_pkg::git_perms::GIT_NO_CHANNEL_BINDING_TOKEN + ); + assert!(error.message.contains("buzz repos bind")); + assert!(error.recovery.is_none()); + + // Unrelated push failures keep the generic code and original text. + let generic = ProjectPullRequestMergeError::from("connection reset".to_string()); + assert_eq!(generic.code, "merge_failed"); + assert_eq!(generic.message, "connection reset"); + } +} diff --git a/desktop/src-tauri/src/commands/project_git_workflow.rs b/desktop/src-tauri/src/commands/project_git_workflow.rs index 39832feb10..624bbf4dfc 100644 --- a/desktop/src-tauri/src/commands/project_git_workflow.rs +++ b/desktop/src-tauri/src/commands/project_git_workflow.rs @@ -32,67 +32,10 @@ pub struct ProjectRepoMergeResult { pub status_publication_error: Option, } -/// Machine-readable recovery metadata for a failed pull-request merge. -#[derive(Debug, Serialize)] -#[serde(rename_all = "camelCase")] -pub struct ProjectPullRequestMergeRecovery { - action: String, - target_branch: String, - source_branch: String, -} - -/// Structured pull-request merge failure returned across the Tauri boundary. -#[derive(Debug, Serialize)] -#[serde(rename_all = "camelCase")] -pub struct ProjectPullRequestMergeError { - code: String, - message: String, - recovery: Option, -} - -impl ProjectPullRequestMergeError { - fn new(code: &str, message: impl Into) -> Self { - Self { - code: code.to_string(), - message: message.into(), - recovery: None, - } - } - - fn conflict(target_branch: String, source_branch: String) -> Self { - Self { - code: "merge_conflict".to_string(), - message: "Pull request has merge conflicts.".to_string(), - recovery: Some(ProjectPullRequestMergeRecovery { - action: "open_terminal".to_string(), - target_branch, - source_branch, - }), - } - } -} - -impl From for ProjectPullRequestMergeError { - fn from(message: String) -> Self { - Self::new("merge_failed", message) - } -} - -fn classify_merge_error( - message: String, - has_conflicts: bool, - target_branch: &str, - source_branch: &str, -) -> ProjectPullRequestMergeError { - if has_conflicts { - ProjectPullRequestMergeError::conflict(target_branch.to_string(), source_branch.to_string()) - } else { - ProjectPullRequestMergeError::new( - "merge_failed", - format!("Pull request merge failed: {message}"), - ) - } -} +/// Machine-readable pull-request merge failure types live in +/// [`super::project_git_merge_error`]; re-imported here for the merge +/// workflow below. +use super::project_git_merge_error::{classify_merge_error, ProjectPullRequestMergeError}; struct ProjectRepoMergeGitResult { message: String, @@ -739,8 +682,8 @@ pub async fn merge_project_pull_request( mod tests { use super::{ align_unborn_head_branch, build_merged_status_event, build_pull_request_status_event, - build_review_request_event, classify_merge_error, normalize_commit, same_repository, - validate_merge_status_metadata, ProjectPullRequestMergeError, + build_review_request_event, normalize_commit, same_repository, + validate_merge_status_metadata, }; use crate::commands::project_git_exec::{build_test_git_auth_config, run_git}; use nostr::{Event, JsonUtil, Keys, Timestamp}; @@ -785,51 +728,6 @@ mod tests { )); } - #[test] - fn merge_conflict_error_has_stable_recovery_metadata() { - let error = - ProjectPullRequestMergeError::conflict("main".to_string(), "feature/demo".to_string()); - - assert_eq!(error.code, "merge_conflict"); - assert_eq!(error.message, "Pull request has merge conflicts."); - let recovery = error.recovery.expect("conflict recovery"); - assert_eq!(recovery.action, "open_terminal"); - assert_eq!(recovery.target_branch, "main"); - assert_eq!(recovery.source_branch, "feature/demo"); - } - - #[test] - fn merge_conflict_error_serializes_for_tauri_clients() { - let error = - ProjectPullRequestMergeError::conflict("main".to_string(), "feature/demo".to_string()); - let value = serde_json::to_value(error).expect("serialize merge conflict"); - - assert_eq!(value["code"], "merge_conflict"); - assert_eq!(value["recovery"]["targetBranch"], "main"); - assert_eq!(value["recovery"]["sourceBranch"], "feature/demo"); - } - - #[test] - fn merge_error_classification_only_recovers_conflicts() { - let conflict = classify_merge_error( - "CONFLICT (content): Merge conflict in src/main.rs".to_string(), - true, - "main", - "feature/demo", - ); - assert_eq!(conflict.code, "merge_conflict"); - assert!(conflict.recovery.is_some()); - - let other = classify_merge_error( - "fatal: refusing to merge unrelated histories".to_string(), - false, - "main", - "feature/demo", - ); - assert_eq!(other.code, "merge_failed"); - assert!(other.recovery.is_none()); - } - #[test] fn merged_status_is_signed_by_repository_owner() { let keys = Keys::generate(); diff --git a/desktop/src/features/projects/lib/projectBranchErrors.test.mjs b/desktop/src/features/projects/lib/projectBranchErrors.test.mjs new file mode 100644 index 0000000000..c1c2a21a28 --- /dev/null +++ b/desktop/src/features/projects/lib/projectBranchErrors.test.mjs @@ -0,0 +1,43 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; + +import { + isNoChannelBindingError, + projectBranchErrorMessage, +} from "./projectBranchErrors.ts"; + +test("recognizes the relay's stable denial token", () => { + // Body produced by the relay push policy (buzz-core + // GIT_NO_CHANNEL_BINDING_BODY), as it arrives wrapped in git stderr. + assert.ok( + isNoChannelBindingError( + "remote: no_channel_binding: repository has no channel binding\nerror: failed to push some refs", + ), + ); +}); + +test("recognizes the legacy spaced phrase from older relays", () => { + assert.ok(isNoChannelBindingError("push denied: no channel binding")); +}); + +test("does not match unrelated errors", () => { + assert.ok(!isNoChannelBindingError("connection reset by peer")); + assert.ok(!isNoChannelBindingError("no channel")); +}); + +test("maps binding denials to remediation copy", () => { + const message = projectBranchErrorMessage( + new Error("remote: no_channel_binding: repository has no channel binding"), + "Failed to create branch.", + ); + assert.ok(message.includes("buzz repos bind")); +}); + +test("passes through other errors and falls back for non-errors", () => { + assert.equal( + projectBranchErrorMessage(new Error("boom"), "fallback"), + "boom", + ); + assert.equal(projectBranchErrorMessage("boom", "fallback"), "fallback"); + assert.equal(projectBranchErrorMessage(null, "fallback"), "fallback"); +}); diff --git a/desktop/src/features/projects/lib/projectBranchErrors.ts b/desktop/src/features/projects/lib/projectBranchErrors.ts new file mode 100644 index 0000000000..2cf990f1af --- /dev/null +++ b/desktop/src/features/projects/lib/projectBranchErrors.ts @@ -0,0 +1,35 @@ +/** + * Relay push-policy denial token for a repository with no `buzz-channel` + * binding. Declared in Rust as `buzz-core::git_perms:: + * GIT_NO_CHANNEL_BINDING_TOKEN`; the relay's denial body starts with it + * ("no_channel_binding: repository has no channel binding"). The legacy + * spaced phrase is kept as a second matcher so this build also recognizes + * denials from relays deployed before the token existed. + */ +const NO_CHANNEL_BINDING_TOKEN = "no_channel_binding"; +const NO_CHANNEL_BINDING_LEGACY_PHRASE = "no channel binding"; + +const NO_CHANNEL_BINDING_COPY = + "This repository is not linked to a project channel, so the relay cannot " + + "authorize access. The repository owner can link it with: " + + "buzz repos bind --id --channel "; + +/** True when a git/relay error text is the unbound-repository denial. */ +export function isNoChannelBindingError(message: string): boolean { + return ( + message.includes(NO_CHANNEL_BINDING_TOKEN) || + message.includes(NO_CHANNEL_BINDING_LEGACY_PHRASE) + ); +} + +/** Map a thrown branch-operation error to user-facing dialog copy. */ +export function projectBranchErrorMessage( + error: unknown, + fallback: string, +): string { + if (!(error instanceof Error)) return fallback; + if (isNoChannelBindingError(error.message)) { + return NO_CHANNEL_BINDING_COPY; + } + return error.message; +} diff --git a/desktop/src/features/projects/ui/ProjectBranchDialogs.tsx b/desktop/src/features/projects/ui/ProjectBranchDialogs.tsx index 20e8d14e1b..c2bde47355 100644 --- a/desktop/src/features/projects/ui/ProjectBranchDialogs.tsx +++ b/desktop/src/features/projects/ui/ProjectBranchDialogs.tsx @@ -5,6 +5,7 @@ import { normalizeProjectBranchName, projectBranchNameError, } from "@/features/projects/lib/projectBranches"; +import { projectBranchErrorMessage } from "@/features/projects/lib/projectBranchErrors"; import { AlertDialog, AlertDialogCancel, @@ -25,14 +26,6 @@ import { } from "@/shared/ui/dialog"; import { Input } from "@/shared/ui/input"; -function errorMessage(error: unknown, fallback: string) { - if (!(error instanceof Error)) return fallback; - if (error.message.includes("no channel binding")) { - return "This repository is owned by another identity and is not linked to a project channel."; - } - return error.message; -} - export function CreateProjectBranchDialog({ existingBranches, onCreate, @@ -69,7 +62,9 @@ export function CreateProjectBranchDialog({ await onCreate(branch); onOpenChange(false); } catch (error) { - setSubmitError(errorMessage(error, "Failed to create branch.")); + setSubmitError( + projectBranchErrorMessage(error, "Failed to create branch."), + ); } } @@ -165,7 +160,9 @@ export function DeleteProjectBranchDialog({ await onDelete(); onOpenChange(false); } catch (error) { - setSubmitError(errorMessage(error, "Failed to delete branch.")); + setSubmitError( + projectBranchErrorMessage(error, "Failed to delete branch."), + ); } } From 63496cc1d4c6f1b7c613801bdcc694169dcf391a Mon Sep 17 00:00:00 2001 From: Tyler <109685178+tlongwell-block@users.noreply.github.com> Date: Wed, 29 Jul 2026 20:35:15 -0400 Subject: [PATCH 30/32] feat(replica): portable heartbeat-token fence with snapshot-local reader routing (#3268) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Expands read-replica usage on the relay per the Rev 2 design (thread 39d1e174 in #buzz-read-only-replica-usage): replaces the Aurora-incompatible WAL-LSN read-side fence observation with a portable **heartbeat token**, makes the freshness proof **snapshot-local to the serving reader session**, and adds a default-off bounded-staleness gate for head fetches. ### Probe (writer side, unchanged ordering) The ordered writer scan is kept verbatim — `S = clock_timestamp()` → masked-visibility `pg_stat_activity` oldest-xact scan — and now ends by committing a heartbeat token **last** on the same pinned connection (single-row `UPDATE replica_heartbeat ... RETURNING token, epoch`, migration `0026`). The single-row update serializes all pods' probes, so tokens are globally commit-ordered; the three-bucket completeness argument carries over unchanged. Ring retains `(token, committed_at, fence_wall)`; epoch change resets the ring; a same-epoch token regression (restore adversary) clears the ring **and rotates the epoch on the writer** so pre-rewind readers fail the epoch check. Cadence 1s. ### Routing (snapshot-local proof) Every routed read opens `BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY` on a reader session and observes the heartbeat as the transaction's **first statement** — the proof's snapshot is exactly the snapshot the page, participants batch, and bridge aux closure read from (`ReadSession` carries the open transaction; drop = rollback). Fail-closed everywhere: begin failure, missing heartbeat row, epoch mismatch, token below the ring, over-budget entry → writer. - **Predicate B (cursor pages, default-on):** same completeness math as the existing fence — cursor timestamp must be ≤ the proved wall; thread candidate-terminal and above-wall pages re-run on the writer. - **Predicate A (head fetches, default-OFF):** gated by `BUZZ_REPLICA_HEAD_MAX_AGE_SECS` (0 = off, clamped to fence staleness). Bounded-stale head semantics are an explicit product decision — **do not enable anywhere without Tyler's backdated-event-semantics acceptance.** ### Observability `buzz_db_route_decision{path, decision, reason}` across all five paths, `buzz_db_replica_heartbeat_age_seconds` gauge, and per-decision debug logs carrying the proved token plus backend identity: `addr:port pid=N`, prefixed with `aurora_db_instance_identifier()` when the endpoint supports it (probed once per process on an autocommit checkout; SQLSTATE 42883 caches a definitive false; identity is evidence, never a routing gate). ## Review & verification - **Wren:** full review 9/9/9 at `5f81b10e5`; identity delta re-review approved at exact head `fedb46368` (90/90 unit, clippy `-D warnings`, fmt independently reproduced). - **Max:** local E2E **PASS** at `5f81b10e5` — isolated PG17 writer + two streaming standbys behind HAProxy; paused-reader legs split exactly as designed (6 replica/fresh + 6 writer/stale), recovery clean, RR snapshot hardening observed in runtime logs. Evidence: `WORK_LOGS/2026-07-28_REPLICA_HEARTBEAT_REPLACEMENT_SHA_E2E.md`. - **My gates at head (same shell):** buzz-db 90 unit + 143 Postgres-gated green (scratch DBs); buzz-relay 767/768 — lone red is the pre-existing `mesh_demo` flake, red on base; clippy `-D warnings` + fmt clean. - Key regression tests: `routed_request_holds_one_snapshot_across_page_and_aux` (mutation-verified both directions), same-epoch rotation, capability-probe negative, head-gate truth table, divergent-fixture routing suite. ## Post-merge plan Merge publishes the immutable `sha-*` main image → bb-public PR pins it → ArgoCD sync → Max runs the production cursor-routing canary on Aurora (positive identity branch proven live). Head gate stays off. Client `since`-widening ships separately. --- ## Update — full read routing (Rev 6, commit `9fa3c9c0b`) Extends routing to the remaining read seams per `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` Rev 6 (thread cf5deba4 in #buzz-read-only-replica-usage). ### New routed seams — ALL deploy-default dark The new seams (`query_events_routed`, `query_events_routed_bounded`, `count_events_routed`, `get_events_by_ids_routed`, `query_feed_{mentions,needs_action,activity}_routed`) are **Bounded-only** and gated on `BUZZ_REPLICA_READ_MAX_AGE_MS`: unset ⇒ every new seam records `writer/disabled` and merging is a no-op. COUNT and feed/by-ids never take the covered arm (deletion visibility: covered bounds insert-completeness only). **Note:** the pre-existing cursor paths (channel windows, thread pages) are *not* gated by this env var — they route at B=0 today and that status quo is unchanged. ### Reader pool (D4/D5) - Lazy pool (`connect_lazy`, `min_connections(0)`): reader-down at boot can't crash the relay; a warn-only boot ping is the only boot-time visibility, and it primes the Aurora identity capability cache. Priming is an optimization only — the routed path itself spends a single acquire budget regardless (see the single-checkout fix below), because a boot ping that *fails* is correlated with exactly the reader-unavailable case the budget bound exists for. - `READER_ACQUIRE_TIMEOUT` = 150ms; a miss fails closed to the writer with reason **`reader_acquire_timeout`** — named for the mechanism, not a diagnosis. The budget includes cold connects and sqlx's `size` counts in-flight dials, so this metric alone does not distinguish contention from slow connection establishment (see `proved_reader` doc-comment for the runbook guidance; the pool gauges are 10s samples and are for capacity planning). - `BUZZ_DB_READ_POOL_SIZE` sizes the reader independently (invalid/0 inherits writer sizing); `read_pool_stats().max` reports the reader's own ceiling. ### Community isolation (formal-model question) Isolation is structural — an explicit `community_id = $n` predicate compiled into every query builder; no RLS, no session-GUC tenant state (zero `CREATE POLICY` across migrations). The `_on` variants reuse the exact same builders with only the executor swapped. Proven by a seven-seam two-community divergent-fixture test (`routed_reads_are_confined_to_the_requested_community`), mutation-tested by Dawn: all four single-predicate stubs killed; the mention-join feeds are defended in depth (three independent predicates) so only complete removal leaks there. ### Accepted limitation D6: client-side staleness on bounded reads (up to `_MS`) is accepted product behavior when the gate is enabled; gate stays off at merge. ## Update — single acquire budget per routed read (commit `dd26caa9f`) Max found (and Dawn independently reproduced, 302–330ms measured) that the boot-unavailable cold path spent **two** stacked `READER_ACQUIRE_TIMEOUT` budgets: the Aurora capability probe did its own `pool.acquire()` before `begin_with` acquired again. Fixed by acquiring **once** per routed read — the capability probe runs on the held connection (`reader_aurora_capability_on`) and the read-only `REPEATABLE READ` transaction begins on that same connection (`Transaction::begin` accepts a `PoolConnection` at `'static` via sqlx's `MaybePoolConnection`). Reason codes unchanged; capability still never negatively cached. Measured routed fallback: ~150ms (one budget). Ships with a PG-gated regression fixture (`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`, authored by Dawn): size-1 reader saturated, capability cache asserted cold, routes through `count_events_routed`, asserts writer answer + one-budget elapsed + `writer/reader_acquire_timeout` label (mutation-tested: `pool_busy` and `reader_validation_error` mutants both killed). It fails at 330ms on the previous commit and passes on this one. ### Verification at `dd26caa9f` (pinned toolchain rustc 1.95.0 via repo `bin/`; ambient rustc 1.89 fails sqlx resolution — always prepend `bin/`) - buzz-db: 94 unit + **150/151** PG-gated serial (`--test-threads=1`; suite is not parallel-safe against one Postgres). The 1 red is `test_usage_metrics_lock_has_single_owner_and_releases_on_drop` — a **pre-existing** same-key-same-database advisory-lock collision with any live relay pointed at the shared `buzz` DB (the relay re-arms `USAGE_METRICS_LOCK_KEY` on a 300s tick). It passes on a private scratch DB with the relay still running — verified this run. Remedy is a scratch `TEST_DATABASE_URL`, **not** terminating lock holders: inspect `pg_locks`/`pg_stat_activity` and resolve the owning process first; an idle advisory holder may be a live dev relay. Pre-existing, not a #3268 blocker — filed as #3619. - buzz-relay: 773/773; clippy `-D warnings` + fmt clean. (`mesh_demo` echo test is a known pre-existing Redis-dependent flake — Dawn confirmed it fails identically on the base commit, untouched by this PR.) - Dawn: independent re-verification at `dd26caa9f` — byte-identical diff application confirmed, suite reproduced, and fixture proven to still discriminate (fails at 320ms with only the production fix reverted). Gate clear. Max: independent rig pass at `dd26caa9f` — 94/94 unit, **15/15 routing/fallback matrix** (cursor behavior, default-dark seams, seven-seam confinement, one-snapshot, dead-reader/no-URL fallback, hard-delete fail-closed, reader max), 773/773 relay, private-DB lock control. Gate clear; recommends merge + staged bb-block rollout (no-reader-URL no-op → cursor-only → observe `buzz_db_route_decision` → conservative nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS`; rollback is config-only). ## Known gap — CI does not run the PG-backed fixtures (#3622) Dawn found post-sign-off (confirmed by Max and Eva at `dd26caa9f`): **every Postgres-backed fixture in this PR is `#[ignore]`d and no CI job selects it.** `Unit Tests` runs `cargo nextest run -p buzz-db --lib` (`Justfile:279-285`, ignored tests excluded); the only `--run-ignored ignored-only` invocations (`ci.yml:690`, `:702`) filter to `relay_invite::tests`. So the one-budget regression, the seven-seam isolation fixture, and the fence/floor-guard/fallback tests have zero automated execution — the verification record above is exhaustive but manual, at this exact SHA. Not introduced by this PR (the `#[ignore]` + narrow-filter pattern predates it; 34 ignored tests total) and not a merge blocker: all new seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set. But it changes the rollout gate — **do not set a nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS` in bb-block until CI enforces these fixtures.** Filed as #3622 (explicit CI selection against the existing backend-integration Postgres archive; ordering note: #3619 must land first or be excluded, since widening the filter would select the colliding usage-metrics lock test). ## Live redteam results (Max, real Helm + PG17 streaming replication at `dd26caa9f`) — #3643, #3644 Max deployed this exact SHA via the repo chart against a physical writer/standby pair and attacked it. **The PR's fenced routing passed every fault**: healthy baseline routes proven on the real standby; paused WAL replay moved head/cursor reads to `writer/stale`; B=0 moved routed reads to `writer/disabled`; reader-URL removal rolled back cleanly. The redteam also found a **pre-existing** production risk this PR does not create and does not fix: NIP-50 search builds its own eager, unfenced pool straight from `READ_DATABASE_URL` (`main.rs:389-402`, introduced by #2084, present in the deployed bb-block `sha-dd222a5` and bb-public `sha-22be8bb` images, and both production ESOs already template `READ_DATABASE_URL`). Measured: ~30s search stall on reader loss, acknowledged-but-unsearchable writes under replica lag (unaffected by `BUZZ_REPLICA_READ_MAX_AGE_MS`), fatal eager connect blocking pod startup when the reader is down, and `replica=true` FTS even at B=0. Filed as **#3643** (fix: pin FTS to the writer, or fence it like the routed seams) with **#3644** for the writer-pointing-reader-URL telemetry gap. Aurora's cluster-ro writer-fallback DNS softens the outage modes in prod but not the lag mode. **Revised ladder consequence:** step 2 ("add `READ_DATABASE_URL`, budget unset") is NOT a no-op with today's binary — it hands FTS an unfenced replica pool. Rollout order is now: merge (still a true no-op — bb-block/bb-public already run the unfenced-FTS code and this PR only adds dark seams) → fix #3643 → CI enforcement #3622 → then reader URL + budget per the original ladder, re-running Max's three deployment tests (reader-down latency, reader-down cold boot, paused-replay read-your-own-write search). ## Activation gate (consolidated) — merge is clear; ALL of the below precede any nonzero `BUZZ_REPLICA_READ_MAX_AGE_MS` Six independent verification passes at `dd26caa9f` (Dawn ×2 incl. byte-level + defang check, Max matrix + live Helm redteam, Wren full CRUD regression + greenfield 0026 + 151/151 PG, Eva). Every induced fault was either handled correctly by this PR's machinery or traced to pre-existing main code. Remaining work gates **activation, not merge**: 1. **#3643** — unfenced NIP-50 FTS pool (pre-existing #2084; live in prod today; also explains the blackhole search hang Wren observed — the routed path was measured bounded at 151-152ms against a silent endpoint, sqlx `inner.rs:252-255`). 2. **#3651** — reader `statement_timeout`: statements after acquire are unbounded; mid-transaction blackhole reproduced hanging ≥15s (Dawn). Small fix via `.after_connect` on the reader pool. 3. **#3622** — CI enforcement of the PG fixtures (with #3619 ordering). 4. **Recovery-conflict cancellation live test** — standby cancels a routed read mid-snapshot → complete writer page, never partial (#3651's fix also bounds the cancellation-never-arrives shape). 5. **DDL replication lag test** — migration on writer + paused replay + budget on → fallback, not client-visible SQL errors. 6. **Reader-tx soak** — 30-60min mixed load; no idle-in-tx accumulation on the standby. Then Max's three deployment tests (reader-down latency, reader-down cold boot, paused-replay read-your-own-write search) re-run on the fixed binary before the first nonzero budget on bb-block. --------- Signed-off-by: Tyler Longwell Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell --- Cargo.lock | 2 + crates/buzz-db/Cargo.toml | 2 + crates/buzz-db/src/event.rs | 40 +- crates/buzz-db/src/feed.rs | 62 +- crates/buzz-db/src/lib.rs | 2362 ++++++++++++++++++++++- crates/buzz-db/src/migration.rs | 19 +- crates/buzz-db/src/replica_fence.rs | 827 ++++++-- crates/buzz-db/src/thread.rs | 55 +- crates/buzz-relay/src/api/bridge.rs | 48 +- crates/buzz-relay/src/config.rs | 118 ++ crates/buzz-relay/src/handlers/count.rs | 16 +- crates/buzz-relay/src/handlers/req.rs | 4 +- crates/buzz-relay/src/main.rs | 14 +- migrations/0026_replica_heartbeat.sql | 38 + schema/schema.sql | 20 + 15 files changed, 3375 insertions(+), 252 deletions(-) create mode 100644 migrations/0026_replica_heartbeat.sql diff --git a/Cargo.lock b/Cargo.lock index c3ea86d6b3..49104b22d2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -949,6 +949,8 @@ dependencies = [ "buzz-core", "chrono", "hex", + "metrics", + "metrics-util", "nostr", "rand 0.10.1", "serde", diff --git a/crates/buzz-db/Cargo.toml b/crates/buzz-db/Cargo.toml index 01f1e172b6..6f76a11bc1 100644 --- a/crates/buzz-db/Cargo.toml +++ b/crates/buzz-db/Cargo.toml @@ -21,6 +21,8 @@ tracing = { workspace = true } thiserror = { workspace = true } nostr = { workspace = true } rand = { workspace = true } +metrics = { workspace = true } [dev-dependencies] tokio = { workspace = true } +metrics-util = { workspace = true } diff --git a/crates/buzz-db/src/event.rs b/crates/buzz-db/src/event.rs index 520fd1536f..0e54196d11 100644 --- a/crates/buzz-db/src/event.rs +++ b/crates/buzz-db/src/event.rs @@ -316,6 +316,17 @@ pub async fn insert_event( /// Uses `QueryBuilder` for dynamic filter composition — avoids string concatenation /// while keeping all user values in bind parameters. pub async fn query_events(pool: &PgPool, q: &EventQuery) -> Result> { + let mut conn = pool.acquire().await?; + query_events_on(&mut conn, q).await +} + +/// [`query_events`] on a specific session — the replica-routing path runs +/// follow-up (aux) queries on the exact reader connection whose heartbeat +/// observation proved coverage for the page they annotate. +pub(crate) async fn query_events_on( + conn: &mut sqlx::PgConnection, + q: &EventQuery, +) -> Result> { // Composite cursor requires both halves. if q.before_id.is_some() && q.until.is_none() { return Err(DbError::InvalidData( @@ -538,7 +549,7 @@ pub async fn query_events(pool: &PgPool, q: &EventQuery) -> Result Result Result { + let mut conn = pool.acquire().await?; + count_events_on(&mut conn, q).await +} + +/// [`count_events`] on a specific session — the replica-routing path runs +/// the count on the exact reader connection whose heartbeat observation +/// proved its predicate. +pub(crate) async fn count_events_on(conn: &mut sqlx::PgConnection, q: &EventQuery) -> Result { // Empty list means "match nothing" — return 0 immediately. if q.kinds.as_deref().is_some_and(|k| k.is_empty()) { return Ok(0); @@ -730,7 +749,7 @@ pub async fn count_events(pool: &PgPool, q: &EventQuery) -> Result { } } - let row = qb.build().fetch_one(pool).await?; + let row = qb.build().fetch_one(&mut *conn).await?; let cnt: i64 = row.try_get("cnt")?; Ok(cnt) @@ -990,6 +1009,21 @@ pub async fn get_events_by_ids( pool: &PgPool, community_id: CommunityId, ids: &[&[u8]], +) -> Result> { + if ids.is_empty() { + return Ok(vec![]); + } + let mut conn = pool.acquire().await?; + get_events_by_ids_on(&mut conn, community_id, ids).await +} + +/// [`get_events_by_ids`] on a specific session — the replica-routing path +/// runs the query on the exact reader connection whose heartbeat +/// observation proved its predicate. +pub(crate) async fn get_events_by_ids_on( + conn: &mut sqlx::PgConnection, + community_id: CommunityId, + ids: &[&[u8]], ) -> Result> { if ids.is_empty() { return Ok(vec![]); @@ -1008,7 +1042,7 @@ pub async fn get_events_by_ids( } qb.push(")"); - let rows = qb.build().fetch_all(pool).await?; + let rows = qb.build().fetch_all(&mut *conn).await?; let mut out = Vec::with_capacity(rows.len()); for row in rows { diff --git a/crates/buzz-db/src/feed.rs b/crates/buzz-db/src/feed.rs index 511a2a6083..40e58d0d06 100644 --- a/crates/buzz-db/src/feed.rs +++ b/crates/buzz-db/src/feed.rs @@ -132,6 +132,29 @@ pub async fn query_mentions( accessible_channel_ids: &[Uuid], since: Option>, limit: i64, +) -> Result> { + let mut conn = pool.acquire().await?; + query_mentions_on( + &mut conn, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await +} + +/// [`query_mentions`] on a specific session — the replica-routing path runs +/// the query on the exact reader connection whose heartbeat observation +/// proved its predicate. +pub(crate) async fn query_mentions_on( + conn: &mut sqlx::PgConnection, + community: CommunityId, + pubkey_bytes: &[u8], + accessible_channel_ids: &[Uuid], + since: Option>, + limit: i64, ) -> Result> { let mut qb = build_mentions_query( community, @@ -140,7 +163,7 @@ pub async fn query_mentions( since, limit, ); - let rows = qb.build().fetch_all(pool).await?; + let rows = qb.build().fetch_all(&mut *conn).await?; collect_stored_events(rows) } @@ -193,6 +216,27 @@ pub async fn query_needs_action( accessible_channel_ids: &[Uuid], since: Option>, limit: i64, +) -> Result> { + let mut conn = pool.acquire().await?; + query_needs_action_on( + &mut conn, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await +} + +/// [`query_needs_action`] on a specific session — see [`query_mentions_on`]. +pub(crate) async fn query_needs_action_on( + conn: &mut sqlx::PgConnection, + community: CommunityId, + pubkey_bytes: &[u8], + accessible_channel_ids: &[Uuid], + since: Option>, + limit: i64, ) -> Result> { let mut qb = build_needs_action_query( community, @@ -201,7 +245,7 @@ pub async fn query_needs_action( since, limit, ); - let rows = qb.build().fetch_all(pool).await?; + let rows = qb.build().fetch_all(&mut *conn).await?; collect_stored_events(rows) } @@ -241,9 +285,21 @@ pub async fn query_activity( accessible_channel_ids: &[Uuid], since: Option>, limit: i64, +) -> Result> { + let mut conn = pool.acquire().await?; + query_activity_on(&mut conn, community, accessible_channel_ids, since, limit).await +} + +/// [`query_activity`] on a specific session — see [`query_mentions_on`]. +pub(crate) async fn query_activity_on( + conn: &mut sqlx::PgConnection, + community: CommunityId, + accessible_channel_ids: &[Uuid], + since: Option>, + limit: i64, ) -> Result> { let mut qb = build_activity_query(community, accessible_channel_ids, since, limit); - let rows = qb.build().fetch_all(pool).await?; + let rows = qb.build().fetch_all(&mut *conn).await?; collect_stored_events(rows) } diff --git a/crates/buzz-db/src/lib.rs b/crates/buzz-db/src/lib.rs index 2a3ba9a63e..0c6ea36dac 100644 --- a/crates/buzz-db/src/lib.rs +++ b/crates/buzz-db/src/lib.rs @@ -181,12 +181,289 @@ pub struct Db { /// route here (see [`Db::read`]); locks, transactions, and anything /// consistency-critical stays on `pool`. pub(crate) read_pool: Option, + /// Maximum connections configured for the read-replica pool (from + /// [`DbConfig::read_max_connections`], defaulting to the writer's + /// sizing). Kept separately from `max_connections` so + /// [`Db::read_pool_stats`] reports the reader's own ceiling — a + /// utilisation gauge derived from the writer's max would understate + /// reader saturation by exactly the ratio of the two pool sizes. + pub(crate) read_max_connections: u32, /// Freshness fence gating cursor-page routing to the replica. /// /// Starts closed; a background probe ([`replica_fence::run_probe`]) - /// advances it after each verified writer→replica LSN handshake. When - /// closed or stale, every cursor page routes to the writer. + /// commits heartbeat tokens and retains proof entries. Routing proves + /// coverage per request on the serving reader session; when the ring is + /// empty or stale, every routed read stays on the writer. pub(crate) fence: std::sync::Arc, + /// Bounded-staleness routing budget `B`: a read routed under + /// [`RoutePredicate::Bounded`] may be served from a proved replica + /// session only when the proved heartbeat entry is at most this old. + /// `None` disables the bounded arm entirely (the rollout default) — + /// bounded-stale read semantics are a product decision, not an + /// invariant, so the gate ships off. + pub(crate) replica_read_max_age: Option, + /// Whether the reader endpoint supports the Aurora PostgreSQL identity + /// function ([`replica_fence::AURORA_IDENTITY_FN`]) — probed + /// once per process on the first routed read (on a plain autocommit + /// checkout, outside any request transaction) and cached. Unset means + /// not yet probed (or the probe hit a transient error and will retry). + /// Shared across `Db` clones. + pub(crate) reader_aurora_identity: std::sync::Arc>, +} + +/// The session that served (or will serve) a routed read, so follow-up +/// queries in the same request (the channel-window aux closure) run on the +/// **same proved snapshot** — a different pooled reader session may sit at a +/// different replay position, and even the same connection advances its +/// snapshot between autocommit statements. +/// +/// `Replica` holds the request's `REPEATABLE READ, READ ONLY` transaction: +/// the heartbeat observation was its first statement, so the snapshot the +/// proof was taken against is exactly the snapshot every follow-up sees. +/// Dropping the session rolls the read-only transaction back and returns +/// the connection to the pool. +/// +/// `Writer` carries the writer pool: follow-ups there are authoritative by +/// construction and need no session pinning. +pub struct ReadSession { + inner: ReadSessionInner, +} + +enum ReadSessionInner { + /// The proved replica request transaction (snapshot-anchored), plus the + /// writer pool so a mid-request replica failure (e.g. a hot-standby + /// recovery conflict cancelling the held snapshot) degrades the session + /// to the writer instead of surfacing an error: degraded capacity, + /// never holes — and never a 500 the writer could have served. + Replica { + tx: sqlx::Transaction<'static, sqlx::Postgres>, + writer: PgPool, + }, + /// The writer pool (cheap clone; Arc-backed). + Writer(PgPool), +} + +impl ReadSession { + /// Query events on this session (see [`Db::query_events`]). + /// + /// If the proved replica transaction fails mid-request, the session + /// permanently degrades to the writer and the query is re-run there. + /// The writer is always at or ahead of any replica replay position, so + /// the degraded follow-up can only observe *more* than the proof-time + /// snapshot, never less — fresher aux rows, the same failure semantics + /// as a request that routed to the writer to begin with. + pub async fn query_events(&mut self, q: &EventQuery) -> Result> { + let degraded = match &mut self.inner { + ReadSessionInner::Replica { tx, writer } => { + match event::query_events_on(tx, q).await { + Ok(rows) => return Ok(rows), + Err(e) => { + tracing::warn!( + error = %e, + "replica session query failed mid-request; degrading to writer" + ); + // Deliberately not a `buzz_db_route_decision` event: + // the page's route was already recorded, and the + // offload metric must stay one-event-per-request. + metrics::counter!("buzz_db_read_session_degraded").increment(1); + writer.clone() + } + } + } + ReadSessionInner::Writer(pool) => return event::query_events(pool, q).await, + }; + // Replacing the inner drops the replica transaction (rolling it + // back and returning the reader connection to its pool). + self.inner = ReadSessionInner::Writer(degraded.clone()); + event::query_events(°raded, q).await + } + + /// Whether this session is a proved replica connection (observability). + pub fn is_replica(&self) -> bool { + matches!(self.inner, ReadSessionInner::Replica { .. }) + } +} + +/// Where one routed read is served (see [`Db::route_read`]). +enum RouteDecision { + /// A reader request transaction whose first-statement heartbeat + /// observation proved this fence entry — the page runs inside it. The + /// `&'static str` is the metric reason (`covered`/`fresh`); the caller + /// records the route only once the page is actually served from the + /// replica, so a post-verification writer re-run or a mid-query replica + /// failure emits exactly one `buzz_db_route_decision` event per request + /// (the offload percentage is read straight off `decision="replica"`). + Replica( + sqlx::Transaction<'static, sqlx::Postgres>, + replica_fence::TokenEntry, + &'static str, + ), + /// Fail closed: serve from the writer pool (already recorded). + Writer, +} + +/// The ONLY place [`route_proof::ChannelScoped`] can be constructed. A +/// crate-root tuple struct would be mintable via `ChannelScoped(())` from +/// every descendant module — tuple-struct field privacy is module-scoped — +/// so the token lives in its own module and E0423 enforces the invariant. +mod route_proof { + use uuid::Uuid; + + /// Proof that a query/page can only return rows with + /// `channel_id IS NOT NULL` — the domain of the commit-time floor guard + /// (migration 0021). `channel_ids` (retains channel-NULL rows) and + /// `global_only = false` are explicitly NOT proofs. + /// + /// Each constructor keys off *how* its path proves channel-bearing-ness: + /// a pinned query filter, a bare `Uuid` argument, or a `NOT NULL` column + /// reached through an inner join. Do not add a universal constructor + /// callers reshape their inputs to fit, and never fabricate a throwaway + /// `EventQuery` purely to mint a token — the proof must be the SQL's + /// shape, not "someone assembled a struct". + #[derive(Clone, Copy)] + pub(crate) struct ChannelScoped(()); + + impl ChannelScoped { + /// Constructor 1: the query pins a single channel + /// (`EventQuery.channel_id = Some(_)`, compiled to a + /// `channel_id = $n` predicate). This proof covers BOTH query + /// builders — the SELECT builder (`event::query_events_on`) and the + /// COUNT builder (`event::count_events`) pin identically; if the + /// two ever drift, this comment is a lie and the routed COUNT seam + /// is unsound. + /// Sound under conjunction: any additional clause (e.g. + /// `channel_ids`, which alone retains channel-NULL rows) is ANDed, + /// and `channel_id = ` never matches NULL — the pin strictly + /// narrows and cannot be widened back out to global rows. + pub(crate) fn from_pinned_channel(q: &crate::event::EventQuery) -> Option { + q.channel_id.map(|_| ChannelScoped(())) + } + + /// Constructor 2 (thread pages): the page is an inner JOIN from + /// `thread_metadata` to `events`, and `thread_metadata.channel_id` + /// is `UUID NOT NULL` — every writer that creates a row passes a + /// concrete channel (`ThreadMetadataParams.channel_id: Uuid`, + /// non-Option). Channel-bearing by construction of the join, not by + /// query predicate. + pub(crate) fn from_thread_metadata_join() -> Self { + ChannelScoped(()) + } + + /// Constructor 3 (channel windows): the channel arrives as a bare + /// `Uuid` argument and the SQL binds it unconditionally + /// (`e.channel_id = $2` in `get_channel_window_on`); every served + /// row is channel-bearing. No `EventQuery` exists on this path. + pub(crate) fn from_channel_id(_channel_id: Uuid) -> Self { + ChannelScoped(()) + } + } +} +use route_proof::ChannelScoped; + +/// The predicate one routed read must satisfy (see [`Db::route_read`]). +/// +/// Discipline: no `Default`, no `Deserialize`, stays non-`pub` — any of +/// those re-opens the [`ChannelScoped`] mint. +enum RoutePredicate { + /// Bounded staleness: the proved entry must be within the configured + /// read budget `B` (default off). Bounds TIME — the page misses at most + /// the freshest `B` of writes. Sound for ANY query shape, including + /// global (channel-NULL) rows: it relies only on heartbeat commit order, + /// not the floor guard. + Bounded, + /// Completeness: the proved wall must cover the page's upper bound. + /// Bounds CONTENT — every row at/below `upper` is present, meaningful + /// even when the cursor is hours old, where `B`-freshness says nothing. + /// Sound ONLY on the floor guard's domain (channel-bearing rows), hence + /// the proof token. `upper` is non-optional: the no-upper-bound + /// post-verifying case is [`RoutePredicate::CoveredPostVerified`]. + /// + /// Bounds INSERT-completeness only — "no missing rows", not "no extra + /// rows". Soft deletes are `UPDATE .. SET deleted_at` commits outside + /// the floor guard and never touch `created_at`, so a covered page can + /// briefly serve a row the writer already excludes; deletion visibility + /// is bounded by replication lag under `FENCE_STALENESS` (30s), not by + /// `upper` or `B`. Do not extend the covered arm to a surface that + /// cannot absorb extra rows (this is why the routed COUNT seam is + /// bounded-only). + Covered { + upper: DateTime, + /// Never read — the field exists so constructing this variant + /// requires minting the token through `route_proof`. + #[allow(dead_code)] + proof: ChannelScoped, + }, + /// Forward-walking thread pages: no upper bound is derivable from the + /// cursor; the caller post-verifies the served rows against the proved + /// wall (full page + tail at/below the wall, else re-run on the writer). + /// Only the thread path constructs this — a general routed caller does + /// no post-verification and must never self-certify. + CoveredPostVerified { + #[allow(dead_code)] + proof: ChannelScoped, + }, + /// Either arm admits, covered tried first (it has no budget dependence). + /// For general routed reads that are channel-pinned AND carry an + /// `until` upper bound. + BoundedOrCovered { + upper: DateTime, + /// Never read — see [`RoutePredicate::Covered::proof`]. + #[allow(dead_code)] + proof: ChannelScoped, + }, +} + +impl RoutePredicate { + /// A channel-window request: cursor pages are covered-only — for deep + /// keyset pages only coverage answers "have all rows below the cursor + /// replayed?" — and a head fetch is bounded. The channel id is the + /// bare-`Uuid` proof that the window SQL pins a channel. + fn from_channel_cursor(channel_id: Uuid, cursor: &Option<(DateTime, Vec)>) -> Self { + match cursor { + Some((ts, _)) => RoutePredicate::Covered { + upper: *ts, + proof: ChannelScoped::from_channel_id(channel_id), + }, + None => RoutePredicate::Bounded, + } + } + + /// General entry point for the routed query seams: derives the strongest + /// sound predicate from the query shape. Never produces a covered arm + /// without both a channel-scope proof AND a real upper bound. + /// + /// `routing_enabled` is whether `BUZZ_REPLICA_READ_MAX_AGE_MS` is set + /// (non-zero). When it is NOT, this returns `Bounded` — which the zero + /// budget then fails closed — so the new seams are genuinely dark at + /// the deploy default even for channel-pinned queries carrying `until`. + /// Without this gate, `BoundedOrCovered` would take the covered arm + /// (which has no budget dependence) and route on day one with no env + /// var set and no kill switch short of removing the replica URL + /// (Dawn's covered-at-zero-budget catch). The pre-existing cursor + /// paths (`Covered`/`CoveredPostVerified` from channel windows and + /// thread pages) intentionally still route at B=0 — status quo, + /// unchanged. + fn for_query(q: &event::EventQuery, routing_enabled: bool) -> Self { + if !routing_enabled { + return RoutePredicate::Bounded; + } + match (ChannelScoped::from_pinned_channel(q), q.until) { + (Some(proof), Some(upper)) => RoutePredicate::BoundedOrCovered { upper, proof }, + _ => RoutePredicate::Bounded, + } + } +} + +/// Map the configured read budget (`BUZZ_REPLICA_READ_MAX_AGE_MS`) to the +/// runtime gate: `0` disables bounded-staleness routing; anything above the +/// fence staleness gate is clamped to it (an entry older than the staleness +/// gate never routes anyway, so a larger budget would only misrepresent the +/// config). +fn read_budget_from_ms(ms: u64) -> Option { + match ms { + 0 => None, + ms => Some(Duration::from_millis(ms).min(replica_fence::FENCE_STALENESS)), + } } /// Snapshot of Postgres connection pool utilisation. @@ -232,6 +509,9 @@ pub struct DbConfig { pub read_database_url: Option, /// Maximum number of connections in the pool. pub max_connections: u32, + /// Maximum connections in the read-replica pool (env + /// `BUZZ_DB_READ_POOL_SIZE`). `None` inherits [`Self::max_connections`]. + pub read_max_connections: Option, /// Minimum number of idle connections to maintain. pub min_connections: u32, /// Seconds to wait when acquiring a connection before timing out. @@ -240,6 +520,13 @@ pub struct DbConfig { pub max_lifetime_secs: u64, /// Seconds a connection may sit idle before being closed. pub idle_timeout_secs: u64, + /// Replica read budget `B` in milliseconds (bounded arm, env + /// `BUZZ_REPLICA_READ_MAX_AGE_MS`). `0` disables bounded-staleness + /// routing — the rollout default. Values above + /// [`replica_fence::FENCE_STALENESS`] are clamped to it: an entry older + /// than the staleness gate never routes anyway, so a larger budget + /// would only misrepresent the config. + pub replica_read_max_age_ms: u64, } impl Default for DbConfig { @@ -251,10 +538,12 @@ impl Default for DbConfig { database_url: "postgres://buzz:buzz_dev@localhost:5432/buzz".to_string(), // sadscan:disable np.postgres.1 read_database_url: None, max_connections: 20, + read_max_connections: None, min_connections: 2, acquire_timeout_secs: 3, max_lifetime_secs: 1800, idle_timeout_secs: 600, + replica_read_max_age_ms: 0, } } } @@ -361,15 +650,22 @@ impl Db { /// proof hold for every insert path that goes through this pool. pub async fn new(config: &DbConfig) -> Result { let pool = Self::connect_pool(config, &config.database_url, true).await?; + let read_max_connections = config + .read_max_connections + .unwrap_or(config.max_connections); let read_pool = match &config.read_database_url { - Some(url) => Some(Self::connect_pool(config, url, false).await?), + Some(url) => Some(Self::connect_read_pool(config, url, read_max_connections)?), None => None, }; + let replica_read_max_age = read_budget_from_ms(config.replica_read_max_age_ms); Ok(Self { pool, max_connections: config.max_connections, read_pool, + read_max_connections, fence: std::sync::Arc::new(replica_fence::ReplicaFence::new()), + replica_read_max_age, + reader_aurora_identity: std::sync::Arc::new(std::sync::OnceLock::new()), }) } @@ -401,13 +697,90 @@ impl Db { Ok(options.connect(url).await?) } + /// Reader acquire timeout — deliberately far below the writer's + /// (seconds-denominated) timeout. Failing closed to the writer must be + /// fast: a saturated reader pool that made routed reads wait the full + /// writer-style timeout would add dead latency during exactly the load + /// spike the offload exists for. A miss here surfaces as + /// `writer/reader_acquire_timeout` (see [`Db::proved_reader`] for why + /// the reason names the mechanism rather than a diagnosis). + const READER_ACQUIRE_TIMEOUT: Duration = Duration::from_millis(150); + + /// Connect the read-replica pool **lazily** — no connection is + /// attempted at construction, so a reader that is down at boot cannot + /// crash the relay (it starts all-writer with the fence closed and + /// recovers when the replica returns). + /// + /// `min_connections` is pinned to 0 explicitly: sqlx's lazy pool still + /// spawns an eager background connect task to satisfy a nonzero + /// minimum, which would reintroduce boot-time reader dial attempts (and + /// their log noise) that "lazy" is meant to avoid. With 0, connections + /// are dialed only on first acquire; the ~10-minute reaper never tops + /// the pool back up, which is fine — routed reads re-fill it on demand. + /// + /// No floor guard: replica sessions are read-only, the trigger never + /// fires there (see [`Db::connect_pool`]). + fn connect_read_pool(config: &DbConfig, url: &str, max_connections: u32) -> Result { + Ok(PgPoolOptions::new() + .max_connections(max_connections) + .min_connections(0) + .acquire_timeout(Self::READER_ACQUIRE_TIMEOUT) + .max_lifetime(Duration::from_secs(config.max_lifetime_secs)) + .idle_timeout(Duration::from_secs(config.idle_timeout_secs)) + .connect_lazy(url)?) + } + + /// Spawn a one-shot reader reachability probe that only WARNs. + /// + /// With a lazy pool and `min_connections(0)`, nothing dials the replica + /// until the first routed read — so a misconfigured `READ_DATABASE_URL` + /// would otherwise be invisible until traffic arrives and quietly falls + /// back to the writer. This ping is the only boot-time reader-down + /// visibility; it must never gate startup or [`Db::spawn_fence_probe`]. + /// + /// On success it also primes the Aurora identity capability cache + /// ([`Db::reader_aurora_identity`]) on the connection it already holds, + /// so the first routed read doesn't spend a second acquire (up to + /// another [`Db::READER_ACQUIRE_TIMEOUT`]) inside + /// [`Db::reader_aurora_capability_on`]. Prime failure is fine: the routed + /// path re-probes on the connection it already holds, so a failed prime + /// costs a round trip rather than a second acquire budget. + pub fn spawn_read_pool_boot_ping(&self) { + let Some(read_pool) = self.read_pool.clone() else { + return; + }; + let aurora_identity = self.reader_aurora_identity.clone(); + tokio::spawn(async move { + match read_pool.acquire().await { + Ok(mut conn) => { + tracing::info!("read replica reachable at boot"); + match replica_fence::reader_supports_aurora_identity(&mut conn).await { + Ok(supported) => { + let _ = aurora_identity.set(supported); + } + Err(e) => tracing::debug!( + error = %e, + "aurora identity boot prime failed; first routed read will probe" + ), + } + } + Err(e) => tracing::warn!( + "read replica unreachable at boot; serving all-writer until it recovers: {e}" + ), + } + }); + } + /// Creates a `Db` from an existing `PgPool` (useful in tests). pub fn from_pool(pool: PgPool) -> Self { Self { max_connections: pool.options().get_max_connections(), + read_max_connections: pool.options().get_max_connections(), pool, read_pool: None, fence: std::sync::Arc::new(replica_fence::ReplicaFence::new()), + replica_read_max_age: None, + reader_aurora_identity: std::sync::Arc::new(std::sync::OnceLock::new()), } } @@ -421,12 +794,21 @@ impl Db { pub fn from_pools(pool: PgPool, read_pool: PgPool) -> Self { Self { max_connections: pool.options().get_max_connections(), + read_max_connections: read_pool.options().get_max_connections(), pool, read_pool: Some(read_pool), fence: std::sync::Arc::new(replica_fence::ReplicaFence::new()), + replica_read_max_age: None, + reader_aurora_identity: std::sync::Arc::new(std::sync::OnceLock::new()), } } + /// Test hook: set the head-fetch routing budget (Predicate A), which + /// [`Db::from_pools`] leaves disabled. + pub fn set_replica_read_max_age_for_tests(&mut self, budget: Option) { + self.replica_read_max_age = budget; + } + /// The freshness fence gating replica routing (see [`replica_fence`]). pub fn fence(&self) -> &std::sync::Arc { &self.fence @@ -438,7 +820,7 @@ impl Db { /// Ordering matters (Perci, PR #2084 review): this must run **after** /// the migration decision. On a relay with `BUZZ_AUTO_MIGRATE` off, the /// writer pool arms the GUC regardless, but if migration 0021 has not - /// been applied there is no trigger enforcing it — and an LSN probe + /// been applied there is no trigger enforcing it — and a heartbeat probe /// would open the fence over an unenforced floor. So the probe is gated /// on an unconditional two-part verification against the live schema: /// catalog shape ([`replica_fence::verify_floor_guard_catalog`]) and @@ -449,14 +831,13 @@ impl Db { /// stays closed: every cursor page routes to the writer. The relay keeps /// serving — degraded capacity, never holes. pub async fn spawn_fence_probe(&self) -> Result { - let Some(read_pool) = &self.read_pool else { + if self.read_pool.is_none() { return Ok(false); - }; + } replica_fence::verify_floor_guard_catalog(&self.pool).await?; replica_fence::verify_floor_guard_behavior(&self.pool).await?; tokio::spawn(replica_fence::run_probe( self.pool.clone(), - read_pool.clone(), std::sync::Arc::clone(&self.fence), )); Ok(true) @@ -465,11 +846,13 @@ impl Db { /// The pool for lag-tolerant reads: the read replica when configured, /// otherwise the writer pool. /// - /// Routing contract — a query may use this pool only when a stale (bounded - /// replication lag) result is acceptable to its caller. Keyset-cursor - /// pagination over immutable history qualifies; head-of-channel fetches, - /// auth/membership checks, locks, and anything inside a transaction do not. - pub fn read(&self) -> &PgPool { + /// Removed as a public escape hatch (Dawn, review of 1b0aa0dfa): the + /// raw replica pool carries **no fence proof**, which is exactly the + /// bug class the routed-read machinery exists to eliminate. All replica + /// reads must go through [`Db::route_read`]-backed entry points; this + /// remains only for the fence's own plumbing tests. + #[cfg(test)] + fn read(&self) -> &PgPool { self.read_pool.as_ref().unwrap_or(&self.pool) } @@ -478,6 +861,153 @@ impl Db { self.read_pool.is_some() } + /// Open a reader request transaction and complete the connection-local + /// half of the fence proof: `BEGIN ISOLATION LEVEL REPEATABLE READ, READ + /// ONLY`, then observe the heartbeat token/epoch as the transaction's + /// **first statement** — anchoring the snapshot every follow-up + /// statement (page, participants, aux closure) sees to exactly the + /// snapshot the proof was taken against — and resolve it against the + /// retained ring. Returns the open transaction together with the + /// strongest [`replica_fence::TokenEntry`] its observation supports, or + /// the fail-closed reason for route metrics. + /// + /// `REPEATABLE READ` is the strongest isolation a hot standby supports + /// (`SERIALIZABLE` is writer-only); `READ ONLY` documents intent and + /// rejects accidental writes. Everything but `Ok` fails closed — begin + /// failure, missing heartbeat row (migration not yet replayed there), + /// observation error, epoch mismatch, or a token below every retained + /// entry all route the request to the writer. + async fn proved_reader( + &self, + read_pool: &PgPool, + ) -> std::result::Result< + ( + sqlx::Transaction<'static, sqlx::Postgres>, + replica_fence::TokenEntry, + ), + &'static str, + > { + // One checkout per routed read. The Aurora capability probe and the + // read-only transaction share a single `acquire()` so the request path + // spends exactly one READER_ACQUIRE_TIMEOUT budget. Probing through + // `read_pool` separately would spend a second budget whenever the + // capability is uncached — i.e. after a failed boot ping, which is + // precisely the reader-unavailable case the bound must hold for. + let conn = match read_pool.acquire().await { + Ok(conn) => conn, + Err(sqlx::Error::PoolTimedOut) => { + tracing::warn!("reader pool acquire timed out; routing to writer"); + return Err("reader_acquire_timeout"); + } + Err(e) => { + tracing::warn!(error = %e, "reader connection acquire failed; routing to writer"); + return Err("reader_validation_error"); + } + }; + let mut conn = conn; + let aurora = self.reader_aurora_capability_on(&mut conn).await; + let mut tx = match sqlx::Transaction::begin( + conn, + Some(sqlx::SqlStr::from_static( + "BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY", + )), + ) + .await + { + Ok(tx) => tx, + // The acquire miss gets its own reason code: the reader pool's + // short acquire timeout (READER_ACQUIRE_TIMEOUT) makes this the + // fast fail-closed path under load, and + // `buzz_db_route_decision{decision="writer",reason="reader_acquire_timeout"}` + // is the operator's alert signal for a struggling reader pool. + // + // The reason deliberately names the mechanism, not a diagnosis: + // `PoolTimedOut` proves only that no connection was handed out + // within the 150ms budget. That budget includes cold connect + // (TCP+TLS+auth), and sqlx's `size` counts in-flight dials, so + // this fires for slow connection establishment as well as for + // established-connection contention — and neither `size == 0` + // nor `size >= max` recovers the missing causal bit (in-flight + // dials hold a size slot, and a cold burst can push + // `active = size - idle` toward max with zero busy connections). + // Runbook: correlate with `buzz_db_read_pool_active` / `_max` + // and reader connection health/latency; high active suggests + // contention, but this metric alone does not distinguish + // contention from slow connects. Note the gauge is a coarse + // sample (BUZZ_POOL_METRICS_INTERVAL_SECS, default 10s) while + // the event it explains lasts ~150ms — a short burst may fall + // between samples entirely, so absence of elevated active is + // NOT evidence of a cold connect. + Err(sqlx::Error::PoolTimedOut) => { + tracing::warn!("reader pool acquire timed out; routing to writer"); + return Err("reader_acquire_timeout"); + } + Err(e) => { + tracing::warn!(error = %e, "reader transaction begin failed; routing to writer"); + return Err("reader_validation_error"); + } + }; + let obs = match replica_fence::observe_heartbeat(&mut tx, aurora).await { + Ok(Some(observation)) => observation, + Ok(None) => return Err("reader_validation_error"), + Err(e) => { + tracing::warn!(error = %e, "heartbeat observation failed; routing to writer"); + return Err("reader_validation_error"); + } + }; + match self.fence.resolve(obs.token, obs.epoch) { + replica_fence::ResolveOutcome::Proved(entry) => { + tracing::debug!( + token = obs.token, + proved_token = entry.token, + backend = %obs.backend, + "reader snapshot proved fence coverage" + ); + Ok((tx, entry)) + } + replica_fence::ResolveOutcome::EpochMismatch => Err("reader_validation_error"), + replica_fence::ResolveOutcome::TokenBehind => Err("reader_token_behind"), + } + } + + /// Whether the reader endpoint supports the Aurora PostgreSQL identity + /// function ([`replica_fence::AURORA_IDENTITY_FN`]), probed + /// once per process and cached (see [`Db::reader_aurora_identity`]). + /// The probe runs on a plain autocommit checkout — never inside the + /// request transaction, where an undefined-function error would abort + /// it. Probe failure (acquire or transient) degrades to the plain + /// identity tuple for THIS request without caching, so a later request + /// retries; identity is evidence, never a routing gate. + /// Aurora capability on a connection the caller already holds, so the + /// routed path never spends a second acquire budget. + async fn reader_aurora_capability_on( + &self, + conn: &mut sqlx::pool::PoolConnection, + ) -> bool { + if let Some(cached) = self.reader_aurora_identity.get() { + return *cached; + } + match replica_fence::reader_supports_aurora_identity(conn).await { + Ok(supported) => *self.reader_aurora_identity.get_or_init(|| supported), + Err(e) => { + tracing::debug!(error = %e, "aurora identity probe failed; will retry"); + false + } + } + } + + /// Record one route decision (Rev 2 observability): which path, where it + /// went, and why. + fn record_route(path: &'static str, decision: &'static str, reason: &'static str) { + metrics::counter!( + "buzz_db_route_decision", + "path" => path, + "decision" => decision, + "reason" => reason, + ) + .increment(1); + } + /// Run pending database migrations. pub async fn migrate(&self) -> Result<()> { migration::run_migrations(&self.pool).await @@ -502,11 +1032,18 @@ impl Db { } /// Pool utilisation stats for the read-replica pool, when configured. + /// + /// `max` is the **reader's** ceiling ([`Db::read_max_connections`]), not + /// the writer's: `buzz_db_read_pool_active / buzz_db_read_pool_max` is + /// the operator's utilisation signal for tuning `BUZZ_DB_READ_POOL_SIZE`, + /// and deriving it from the writer's max would misreport saturation by + /// exactly the ratio of the two pool sizes — in the direction that hides + /// the problem. pub fn read_pool_stats(&self) -> Option { self.read_pool.as_ref().map(|p| DbPoolStats { size: p.size(), idle: p.num_idle() as u32, - max: self.max_connections, + max: self.read_max_connections, }) } @@ -1094,15 +1631,126 @@ impl Db { } /// Queries events matching the given filter parameters. + /// + /// Always reads from the WRITER pool. If the result influences a write + /// or a permission decision, this is the method to call. Display-path + /// callers that tolerate bounded staleness should use + /// [`Db::query_events_routed`] instead — converting a caller is an + /// explicit, per-callsite decision, never a change to this method. pub async fn query_events(&self, q: &EventQuery) -> Result> { event::query_events(&self.pool, q).await } + /// [`Db::query_events`] with replica routing — the opt-in fast path for + /// display reads. + /// + /// Rule of thumb: **if the result influences a write or a permission, + /// it reads from the writer** — do not convert such a caller to this + /// method. Every new caller must be added to the caller-classification + /// table in `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md`. + /// + /// Routing derives the strongest sound predicate from the query shape + /// ([`RoutePredicate::for_query`]): a channel-pinned query with an + /// `until` upper bound may be served covered (provably complete below + /// the fence wall); anything else is bounded-staleness only. The whole + /// seam is gated on `BUZZ_REPLICA_READ_MAX_AGE_MS` (default off): when + /// unset, even covered-eligible queries stay on the writer, so merging + /// this seam is a true no-op until the budget is configured. Every + /// failure fails closed to the writer. + pub async fn query_events_routed( + &self, + path: &'static str, + q: &EventQuery, + ) -> Result> { + let predicate = RoutePredicate::for_query(q, self.replica_read_max_age.is_some()); + match self.route_read(path, predicate).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match event::query_events_on(&mut tx, q).await { + Ok(events) => { + Self::record_route(path, "replica", reason); + Ok(events) + } + Err(e) => { + // Mid-query replica failure: fail closed to the + // writer rather than surfacing a routed error. + tracing::warn!(path, "replica read failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + event::query_events(&self.pool, q).await + } + } + } + RouteDecision::Writer => event::query_events(&self.pool, q).await, + } + } + + /// [`Db::query_events_routed`] restricted to the BOUNDED arm — for + /// reads whose result feeds a COUNT rather than a displayed page. + /// + /// The covered arm bounds insert-completeness only; stale deletions can + /// briefly inflate the result set (see [`RoutePredicate::Covered`]). A + /// display page absorbs that per-row; a number derived from the rows + /// does not. Same classification-table requirement as + /// [`Db::query_events_routed`]. + pub async fn query_events_routed_bounded( + &self, + path: &'static str, + q: &EventQuery, + ) -> Result> { + match self.route_read(path, RoutePredicate::Bounded).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match event::query_events_on(&mut tx, q).await { + Ok(events) => { + Self::record_route(path, "replica", reason); + Ok(events) + } + Err(e) => { + tracing::warn!(path, "replica read failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + event::query_events(&self.pool, q).await + } + } + } + RouteDecision::Writer => event::query_events(&self.pool, q).await, + } + } + /// Count events matching the given query (NIP-45 COUNT support). + /// + /// Always reads from the WRITER pool — see [`Db::query_events`] for the + /// writer-vs-routed rule. pub async fn count_events(&self, q: &EventQuery) -> Result { event::count_events(&self.pool, q).await } + /// [`Db::count_events`] with replica routing — same contract, rules, + /// and classification-table requirement as [`Db::query_events_routed`]. + /// + /// Counts route on the BOUNDED arm only, never covered: the covered + /// arm bounds insert-completeness but not deletion visibility (soft + /// deletes are UPDATEs outside the floor guard), and a count has no + /// downstream per-row re-filter to absorb extra rows — a silently + /// inflated number for up to `FENCE_STALENESS` is a different product + /// statement than a page briefly showing a deleted row. `Bounded` ties + /// the error to the accepted budget `B`. + pub async fn count_events_routed(&self, path: &'static str, q: &EventQuery) -> Result { + match self.route_read(path, RoutePredicate::Bounded).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match event::count_events_on(&mut tx, q).await { + Ok(count) => { + Self::record_route(path, "replica", reason); + Ok(count) + } + Err(e) => { + tracing::warn!(path, "replica count failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + event::count_events(&self.pool, q).await + } + } + } + RouteDecision::Writer => event::count_events(&self.pool, q).await, + } + } + /// Return whether a creator-signed huddle-start event links a parent /// channel to an ephemeral huddle channel. pub async fn huddle_started_link_exists( @@ -1222,6 +1870,37 @@ impl Db { event::get_events_by_ids(&self.pool, community_id, ids).await } + /// [`Db::get_events_by_ids`] with replica routing — same contract and + /// classification-table requirement as [`Db::query_events_routed`]. + /// + /// By-id fetches route on the BOUNDED arm only: an id list carries no + /// channel pin, so no fence floor can prove insert-completeness — the + /// covered arm is structurally unavailable. Used for FTS hit hydration, + /// where a missing row degrades to a skipped search hit downstream. + pub async fn get_events_by_ids_routed( + &self, + path: &'static str, + community_id: CommunityId, + ids: &[&[u8]], + ) -> Result> { + match self.route_read(path, RoutePredicate::Bounded).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match event::get_events_by_ids_on(&mut tx, community_id, ids).await { + Ok(events) => { + Self::record_route(path, "replica", reason); + Ok(events) + } + Err(e) => { + tracing::warn!(path, "replica read failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + event::get_events_by_ids(&self.pool, community_id, ids).await + } + } + } + RouteDecision::Writer => event::get_events_by_ids(&self.pool, community_id, ids).await, + } + } + /// Exclusively claim a batch of due matcher jobs from one community. pub async fn claim_due_push_match_batch( &self, @@ -1990,19 +2669,25 @@ impl Db { /// Fetch replies under a root event. /// - /// Routing: the head fetch (`cursor: None`) always reads the writer. - /// Cursor-bearing pages may read the replica pool when one is configured - /// AND the freshness fence is open ([`replica_fence`]). Thread pagination - /// walks forward from oldest to newest, so a replica page is served only - /// when it is provably complete: + /// Routing mirrors [`Db::get_channel_window_with_session`]: a head + /// fetch (`cursor: None`) is Predicate A (bounded staleness, gated by + /// the default-off head budget); cursor pages are Predicate B + /// (completeness). Thread pagination walks **forward** from oldest to + /// newest, so a cursor carries no upper bound — instead the served page + /// is post-verified against the wall the serving session proved: /// /// - an under-`limit` page is a candidate terminal page — the client /// treats it as EOF, so it is re-run on the writer to keep the EOF /// decision authoritative (a lagged replica could truncate the tail); - /// - a full page whose newest row exceeds the fence could straddle a row - /// the replica has not replayed (commit order is not `created_at` - /// order), so it is also re-run on the writer. Only a full page that - /// sits entirely at or below the fence is served from the replica. + /// - a full page whose newest row exceeds the proved fence wall could + /// straddle a row the session has not replayed (commit order is not + /// `created_at` order), so it is also re-run on the writer. Only a + /// full page that sits entirely at or below the proved wall is served + /// from the replica. + /// + /// A head fetch routed under Predicate A skips the re-run: bounded + /// staleness (missing at most the freshest budget-window of replies) is + /// exactly the semantic the head gate accepts. pub async fn get_thread_replies( &self, community_id: CommunityId, @@ -2011,25 +2696,59 @@ impl Db { limit: u32, cursor: Option<&[u8]>, ) -> Result> { - if cursor.is_some() && self.has_read_pool() && self.fence.verified_through().is_some() { - let replies = thread::get_thread_replies( - self.read(), + let (path, predicate): (&'static str, RoutePredicate) = match cursor { + Some(_) => ( + "thread_cursor", + RoutePredicate::CoveredPostVerified { + proof: ChannelScoped::from_thread_metadata_join(), + }, + ), + None => ("thread_head", RoutePredicate::Bounded), + }; + if let RouteDecision::Replica(mut tx, entry, reason) = + self.route_read(path, predicate).await + { + match thread::get_thread_replies_on( + &mut tx, community_id, root_event_id, depth_limit, limit, cursor, ) - .await?; - let full = replies.len() >= limit as usize; - let below_fence = replies - .last() - .is_some_and(|tail| self.fence.covers(tail.created_at)); - if full && below_fence { - return Ok(replies); + .await + { + Ok(replies) => { + if cursor.is_none() { + // Predicate A: bounded-stale head page, served as proved. + Self::record_route(path, "replica", reason); + return Ok(replies); + } + let full = replies.len() >= limit as usize; + let below_fence = replies + .last() + .is_some_and(|tail| tail.created_at <= entry.fence_wall); + if full && below_fence { + Self::record_route(path, "replica", reason); + return Ok(replies); + } + // Candidate terminal page, or page reaching above the + // proved wall — verify against the writer. Recorded as + // the request's ONLY route event: the replica leg was + // discarded, so counting it would overstate offload. + Self::record_route("thread_eof", "writer", "stale"); + } + Err(e) => { + // Mid-request replica failure (e.g. a hot-standby + // recovery conflict) fails closed to the writer. + tracing::warn!( + error = %e, + path, + "replica thread query failed; re-running on writer" + ); + Self::record_route(path, "writer", "replica_error"); + } } - // Candidate terminal page, or page reaching above the fence — - // verify against the writer. } thread::get_thread_replies( &self.pool, @@ -2053,15 +2772,8 @@ impl Db { /// One channel window: top-level rows + summaries + server `has_more`. /// - /// Routing: the head fetch (`cursor: None`) always reads the writer — it - /// must include just-committed events. A cursor-bearing page scrolls - /// *backward* into history bounded above by the cursor timestamp - /// (`created_at < ts`, or `= ts` with the id tiebreak), so it may read - /// the replica when one is configured AND the freshness fence covers the - /// cursor timestamp: every row the page could contain is then provably - /// replayed on the replica ([`replica_fence`]). Pages whose cursor - /// reaches above the fence — the freshest sliver of history — stay on - /// the writer. + /// Convenience wrapper over [`Db::get_channel_window_with_session`] for + /// callers with no follow-up queries; the serving session is released. pub async fn get_channel_window( &self, community_id: CommunityId, @@ -2070,11 +2782,199 @@ impl Db { cursor: Option<(DateTime, Vec)>, kind_filter: Option<&[u32]>, ) -> Result { - let pool = match &cursor { - Some((ts, _)) if self.has_read_pool() && self.fence.covers(*ts) => self.read(), - _ => &self.pool, + self.get_channel_window_with_session(community_id, channel_id, limit, cursor, kind_filter) + .await + .map(|(window, _session)| window) + } + + /// [`Db::get_channel_window`], additionally returning the session that + /// served the page so request-scoped follow-ups (the aux closure) run on + /// the same proved connection. + /// + /// Routing: + /// + /// - **Cursor page** (Predicate B — completeness): scrolls *backward* + /// into history bounded above by the cursor timestamp (`created_at < + /// ts`, or `= ts` with the id tiebreak), so it may be served by a + /// replica session when one is configured AND that session **proves** + /// coverage of the cursor timestamp: the heartbeat token/epoch is + /// observed on the exact connection that will serve the page and + /// resolved against the fence's retained ring ([`replica_fence`]). + /// - **Head fetch** (Predicate A — bounded staleness): served by a + /// proved replica session only when the head gate is configured + /// ([`DbConfig::replica_read_max_age_ms`], default off) and the + /// proved entry is within the budget. This trades a bounded staleness + /// window (budget plus probe cadence) on the GET leg for writer + /// offload. NOTE: enabling the budget also breaks read-your-own-writes + /// on the GET leg; the client-side WS `since`-overlap union intended + /// to cover fresh events has NOT shipped yet — do not enable + /// `BUZZ_REPLICA_HEAD_MAX_AGE_SECS` until it has, proven by a + /// post-then-immediately-refetch test. + /// + /// Every failure fails closed to the writer and is recorded in + /// `buzz_db_route_decision`. + pub async fn get_channel_window_with_session( + &self, + community_id: CommunityId, + channel_id: Uuid, + limit: u32, + cursor: Option<(DateTime, Vec)>, + kind_filter: Option<&[u32]>, + ) -> Result<(thread::ChannelWindow, ReadSession)> { + let path: &'static str = if cursor.is_some() { + "channel_cursor" + } else { + "channel_head" + }; + match self + .route_read( + path, + RoutePredicate::from_channel_cursor(channel_id, &cursor), + ) + .await + { + RouteDecision::Replica(mut tx, _entry, reason) => { + match thread::get_channel_window_on( + &mut tx, + community_id, + channel_id, + limit, + cursor.clone(), + kind_filter, + ) + .await + { + Ok(window) => { + Self::record_route(path, "replica", reason); + return Ok(( + window, + ReadSession { + inner: ReadSessionInner::Replica { + tx, + writer: self.pool.clone(), + }, + }, + )); + } + Err(e) => { + // A mid-request replica failure (e.g. a hot-standby + // recovery conflict cancelling the held snapshot) + // fails closed to the writer: a stale-but-served + // page, never an error the writer could have + // answered. Dropping `tx` rolls the reader + // transaction back. + tracing::warn!( + error = %e, + path, + "replica window query failed; re-running on writer" + ); + Self::record_route(path, "writer", "replica_error"); + } + } + } + RouteDecision::Writer => {} + } + let window = thread::get_channel_window( + &self.pool, + community_id, + channel_id, + limit, + cursor, + kind_filter, + ) + .await?; + Ok(( + window, + ReadSession { + inner: ReadSessionInner::Writer(self.pool.clone()), + }, + )) + } + + /// Shared route decision for one read: evaluate the predicate against a + /// proved reader session and record the decision. Fail closed to the + /// writer everywhere. + async fn route_read(&self, path: &'static str, predicate: RoutePredicate) -> RouteDecision { + let Some(read_pool) = &self.read_pool else { + Self::record_route(path, "writer", "disabled"); + return RouteDecision::Writer; + }; + // Cheap prechecks on the shared ring before spending a reader + // checkout; the connection-local observation still has to prove it. + let Some(newest) = self.fence.newest() else { + Self::record_route(path, "writer", "uninitialized"); + return RouteDecision::Writer; + }; + // Precheck helpers against the newest shared entry: if the newest + // cannot satisfy an arm, no proved (older-or-equal) entry can. + let bounded_precheck = + |budget: &Option| -> std::result::Result<(), &'static str> { + match budget { + Some(budget) if newest.committed_at.elapsed() <= *budget => Ok(()), + Some(_) => Err("stale"), + None => Err("disabled"), + } + }; + let covered_precheck = |upper: &DateTime| -> std::result::Result<(), &'static str> { + if *upper <= newest.fence_wall { + Ok(()) + } else { + Err("stale") + } + }; + let precheck = match &predicate { + RoutePredicate::Bounded => bounded_precheck(&self.replica_read_max_age), + RoutePredicate::Covered { upper, .. } => covered_precheck(upper), + // No upper bound: the caller post-verifies served rows. + RoutePredicate::CoveredPostVerified { .. } => Ok(()), + // Covered first (no budget dependence), else bounded. + RoutePredicate::BoundedOrCovered { upper, .. } => { + covered_precheck(upper).or_else(|_| bounded_precheck(&self.replica_read_max_age)) + } }; - thread::get_channel_window(pool, community_id, channel_id, limit, cursor, kind_filter).await + if let Err(reason) = precheck { + Self::record_route(path, "writer", reason); + return RouteDecision::Writer; + } + match self.proved_reader(read_pool).await { + Ok((tx, entry)) => { + // Re-evaluate against the entry the session actually proved + // (it may be older than the shared newest). + let bounded_holds = || { + self.replica_read_max_age + .is_some_and(|budget| entry.committed_at.elapsed() <= budget) + }; + let verdict: Option<&'static str> = match &predicate { + RoutePredicate::Bounded => bounded_holds().then_some("fresh"), + RoutePredicate::Covered { upper, .. } => { + (*upper <= entry.fence_wall).then_some("covered") + } + // No upper bound: the caller post-verifies the served + // rows against the proved wall. + RoutePredicate::CoveredPostVerified { .. } => Some("covered"), + RoutePredicate::BoundedOrCovered { upper, .. } => { + if *upper <= entry.fence_wall { + Some("covered") + } else { + bounded_holds().then_some("fresh") + } + } + }; + match verdict { + Some(reason) => RouteDecision::Replica(tx, entry, reason), + None => { + // The session proves an older entry than the + // predicate needs (replication lag) — fail closed. + Self::record_route(path, "writer", "stale"); + RouteDecision::Writer + } + } + } + Err(reason) => { + Self::record_route(path, "writer", reason); + RouteDecision::Writer + } + } } /// Look up a single thread_metadata row by event_id. @@ -2239,6 +3139,67 @@ impl Db { .await } + /// [`Db::query_feed_mentions`] with replica routing — same contract and + /// classification-table requirement as [`Db::query_events_routed`]. + /// + /// Feed queries route on the BOUNDED arm only: the `accessible_channel_ids` + /// parameter admits community-global rows alongside channel rows, so no + /// single channel's fence floor can prove completeness — the covered arm + /// is structurally unavailable, not merely unchosen. + pub async fn query_feed_mentions_routed( + &self, + path: &'static str, + community: CommunityId, + pubkey_bytes: &[u8], + accessible_channel_ids: &[Uuid], + since: Option>, + limit: i64, + ) -> Result> { + match self.route_read(path, RoutePredicate::Bounded).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match feed::query_mentions_on( + &mut tx, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await + { + Ok(events) => { + Self::record_route(path, "replica", reason); + Ok(events) + } + Err(e) => { + tracing::warn!(path, "replica read failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + feed::query_mentions( + &self.pool, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await + } + } + } + RouteDecision::Writer => { + feed::query_mentions( + &self.pool, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await + } + } + } + /// Find events that require action from the given pubkey. pub async fn query_feed_needs_action( &self, @@ -2259,6 +3220,63 @@ impl Db { .await } + /// [`Db::query_feed_needs_action`] with replica routing — BOUNDED arm + /// only; see [`Db::query_feed_mentions_routed`] for why the covered arm + /// is structurally unavailable to feed queries. + pub async fn query_feed_needs_action_routed( + &self, + path: &'static str, + community: CommunityId, + pubkey_bytes: &[u8], + accessible_channel_ids: &[Uuid], + since: Option>, + limit: i64, + ) -> Result> { + match self.route_read(path, RoutePredicate::Bounded).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match feed::query_needs_action_on( + &mut tx, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await + { + Ok(events) => { + Self::record_route(path, "replica", reason); + Ok(events) + } + Err(e) => { + tracing::warn!(path, "replica read failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + feed::query_needs_action( + &self.pool, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await + } + } + } + RouteDecision::Writer => { + feed::query_needs_action( + &self.pool, + community, + pubkey_bytes, + accessible_channel_ids, + since, + limit, + ) + .await + } + } + } + /// Find recent activity across accessible channels. pub async fn query_feed_activity( &self, @@ -2270,6 +3288,53 @@ impl Db { feed::query_activity(&self.pool, community, accessible_channel_ids, since, limit).await } + /// [`Db::query_feed_activity`] with replica routing — BOUNDED arm only; + /// see [`Db::query_feed_mentions_routed`] for why the covered arm is + /// structurally unavailable to feed queries. + pub async fn query_feed_activity_routed( + &self, + path: &'static str, + community: CommunityId, + accessible_channel_ids: &[Uuid], + since: Option>, + limit: i64, + ) -> Result> { + match self.route_read(path, RoutePredicate::Bounded).await { + RouteDecision::Replica(mut tx, _entry, reason) => { + match feed::query_activity_on( + &mut tx, + community, + accessible_channel_ids, + since, + limit, + ) + .await + { + Ok(events) => { + Self::record_route(path, "replica", reason); + Ok(events) + } + Err(e) => { + tracing::warn!(path, "replica read failed; re-running on writer: {e}"); + Self::record_route(path, "writer", "replica_error"); + feed::query_activity( + &self.pool, + community, + accessible_channel_ids, + since, + limit, + ) + .await + } + } + } + RouteDecision::Writer => { + feed::query_activity(&self.pool, community, accessible_channel_ids, since, limit) + .await + } + } + } + /// Create a new API token record. #[allow(clippy::too_many_arguments)] pub async fn create_api_token( @@ -5436,26 +6501,183 @@ mod tests { assert!(db.read_pool_stats().is_none()); } - /// Channel window: head fetch (no cursor) reads the WRITER; cursor pages - /// read the REPLICA. Divergent fixtures prove which pool served each. - #[tokio::test] - #[ignore = "requires Postgres"] - async fn channel_window_routes_head_to_writer_and_cursor_pages_to_replica() { - let admin = PgPool::connect(&admin_url().await) - .await - .expect("connect admin"); - let (writer, wname) = create_scratch_db(&admin, "routing_w").await; - let (replica, rname) = create_scratch_db(&admin, "routing_r").await; + #[test] + fn read_budget_zero_disables_and_large_values_clamp_to_staleness() { + assert_eq!(read_budget_from_ms(0), None, "0 = bounded routing off"); + assert_eq!( + read_budget_from_ms(1000), + Some(std::time::Duration::from_millis(1000)) + ); + assert_eq!( + read_budget_from_ms(10_000_000), + Some(replica_fence::FENCE_STALENESS), + "budgets above the staleness gate clamp to it" + ); + } - let author = nostr::Keys::generate(); - let community = Uuid::new_v4(); + /// Truth table for [`RoutePredicate::for_query`]: the strongest sound + /// predicate per query shape, and — the deploy-day default row — that + /// `routing_enabled = false` (BUZZ_REPLICA_READ_MAX_AGE_MS unset) + /// forces `Bounded` even for covered-eligible shapes, so the zero + /// budget fails the new seams closed (Dawn's covered-at-zero-budget + /// catch, design doc rev 5). + #[test] + fn for_query_predicate_truth_table() { + let community = CommunityId::from_uuid(Uuid::new_v4()); let channel = Uuid::new_v4(); - seed_community_channel(&writer, community, channel, &author).await; - seed_community_channel(&replica, community, channel, &author).await; + let until = chrono::Utc::now(); - // Shared history (both databases): m1 < m2 < m3. - let base = 1_700_000_000u64; - let m1 = signed_event_at(&author, "m1", base); + let pinned_with_until = { + let mut q = event::EventQuery::for_community(community); + q.channel_id = Some(channel); + q.until = Some(until); + q + }; + let pinned_no_until = { + let mut q = event::EventQuery::for_community(community); + q.channel_id = Some(channel); + q + }; + let unpinned_with_until = { + let mut q = event::EventQuery::for_community(community); + q.until = Some(until); + q + }; + let global_only = { + let mut q = event::EventQuery::for_community(community); + q.global_only = true; + q.until = Some(until); + q + }; + + // Deploy-day default: budget unset ⇒ Bounded regardless of shape. + // The zero budget then fails Bounded closed, so the new seams + // record writer/disabled — merging with no env var set is a no-op. + assert!( + matches!( + RoutePredicate::for_query(&pinned_with_until, false), + RoutePredicate::Bounded + ), + "budget unset must not reach the covered arm even when eligible" + ); + + // Budget set + channel pin + until ⇒ the strongest predicate. + assert!(matches!( + RoutePredicate::for_query(&pinned_with_until, true), + RoutePredicate::BoundedOrCovered { .. } + )); + + // Missing either covered precondition ⇒ Bounded. + assert!(matches!( + RoutePredicate::for_query(&pinned_no_until, true), + RoutePredicate::Bounded + )); + assert!(matches!( + RoutePredicate::for_query(&unpinned_with_until, true), + RoutePredicate::Bounded + )); + // global_only implies `channel_id = None`, so the channel-pin + // precondition fails and no covered arm is possible — `for_query` + // never inspects `global_only` itself; the row holds because + // constructor 1 (channel pin) returns None for an unpinned query. + assert!(matches!( + RoutePredicate::for_query(&global_only, true), + RoutePredicate::Bounded + )); + } + + /// The pre-existing cursor paths are NOT budget-gated: a channel-window + /// cursor page still derives `Covered` with no `routing_enabled` input + /// at all — at B=0 today it routes covered, and that status quo is + /// intentionally unchanged by the `for_query` gate (Max's matrix row: + /// old paths route at budget-unset; only the new seams go dark). + #[test] + fn channel_cursor_predicate_is_not_budget_gated() { + let channel = Uuid::new_v4(); + let cursor = Some((chrono::Utc::now(), vec![1u8; 32])); + assert!(matches!( + RoutePredicate::from_channel_cursor(channel, &cursor), + RoutePredicate::Covered { .. } + )); + // Head fetch (no cursor) is bounded — gated by the budget. + assert!(matches!( + RoutePredicate::from_channel_cursor(channel, &None), + RoutePredicate::Bounded + )); + } + + /// D5 wiring: `read_pool_stats().max` must be the READER pool's own + /// ceiling, not the writer's — `buzz_db_read_pool_active / _max` is the + /// operator's utilisation signal and inheriting the writer's max hides + /// reader saturation by exactly the sizing ratio. Pure wiring test: + /// `connect_lazy` never touches the network, but it does spawn the + /// pool reaper task, which needs a Tokio runtime — hence + /// `#[tokio::test]` despite the test body itself never awaiting. + #[tokio::test] + async fn read_pool_stats_reports_reader_ceiling_not_writer() { + let writer = sqlx::postgres::PgPoolOptions::new() + .max_connections(20) + .connect_lazy(TEST_DB_URL) + .expect("lazy writer pool"); + let reader = sqlx::postgres::PgPoolOptions::new() + .max_connections(40) + .connect_lazy(TEST_DB_URL) + .expect("lazy reader pool"); + let db = Db::from_pools(writer, reader); + assert_eq!(db.pool_stats().max, 20); + assert_eq!( + db.read_pool_stats().expect("read pool configured").max, + 40, + "reader gauge must report the reader's own ceiling" + ); + } + + /// D4 wiring: the reader pool is built lazily with `min_connections(0)` + /// and the short reader acquire timeout — construction must succeed + /// with no replica listening (reader-down at boot must not crash the + /// relay), and `read_max_connections` must honour + /// `DbConfig::read_max_connections` over the writer sizing. + /// `#[tokio::test]` because `connect_lazy` spawns the pool reaper task, + /// which needs a Tokio runtime even though nothing is dialed. + #[tokio::test] + async fn connect_read_pool_is_lazy_and_independently_sized() { + let config = DbConfig { + max_connections: 20, + read_max_connections: Some(7), + ..DbConfig::default() + }; + // Unroutable per RFC 5737 TEST-NET-1: proves nothing is dialed at + // construction time. + let pool = Db::connect_read_pool(&config, "postgres://user:pw@192.0.2.1:5432/none", 7) + .expect("lazy construction must not dial the replica"); + assert_eq!(pool.options().get_max_connections(), 7); + assert_eq!(pool.options().get_min_connections(), 0); + assert_eq!( + pool.options().get_acquire_timeout(), + Db::READER_ACQUIRE_TIMEOUT + ); + } + + /// Channel window: head fetch (no cursor) reads the WRITER; cursor pages + /// read the REPLICA. Divergent fixtures prove which pool served each. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn channel_window_routes_head_to_writer_and_cursor_pages_to_replica() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "routing_w").await; + let (replica, rname) = create_scratch_db(&admin, "routing_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + // Shared history (both databases): m1 < m2 < m3. + let base = 1_700_000_000u64; + let m1 = signed_event_at(&author, "m1", base); let m2 = signed_event_at(&author, "m2", base + 10); let m3 = signed_event_at(&author, "m3", base + 20); for pool in [&writer, &replica] { @@ -5519,6 +6741,984 @@ mod tests { drop_scratch_db(&admin, writer, &wname).await; } + /// Fail-closed on a mid-request replica failure (Dawn, review of + /// 1b0aa0dfa): a replica-routed page whose query errors *after* the + /// proof (the live shape is a hot-standby recovery conflict — 40001 / + /// 25P02 — cancelling the held snapshot under `max_standby_streaming_delay`) + /// must be re-run on the writer and served, never surfaced as an error + /// the writer could have answered. Degraded capacity, never holes. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn replica_window_failure_falls_back_to_writer() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "fb_w").await; + let (replica, rname) = create_scratch_db(&admin, "fb_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + let m1 = signed_event_at(&author, "m1", base); + let m2 = signed_event_at(&author, "m2", base + 10); + let m3 = signed_event_at(&author, "m3", base + 20); + for pool in [&writer, &replica] { + for ev in [&m1, &m2, &m3] { + insert_top_level(pool, community, channel, ev).await; + } + } + let marker = signed_event_at(&author, "replica-only-marker", base + 5); + insert_top_level(&replica, community, channel, &marker).await; + + let db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + + let head = db + .get_channel_window(cid, channel, 1, None, None) + .await + .expect("head window"); + let cursor = head.next_cursor.expect("has_more implies next_cursor"); + + // Guard against a vacuous pass: the cursor page must actually be + // replica-eligible before we break the replica. + let healthy = db + .get_channel_window(cid, channel, 10, Some(cursor.clone()), None) + .await + .expect("healthy cursor window"); + assert!( + healthy + .rows + .iter() + .any(|r| r.stored_event.event.content == "replica-only-marker"), + "fixture must route the cursor page to the replica while healthy" + ); + + // Break the replica AFTER the proof point: the heartbeat table stays + // intact (the observation succeeds), the page query then fails. + sqlx::query("DROP TABLE events CASCADE") + .execute(&replica) + .await + .expect("drop replica events"); + + let page = db + .get_channel_window(cid, channel, 10, Some(cursor), None) + .await + .expect("replica failure must fall back to the writer, not error"); + let contents: Vec<&str> = page + .rows + .iter() + .map(|r| r.stored_event.event.content.as_str()) + .collect(); + assert_eq!( + contents, + vec!["m2", "m1"], + "fallback page must be the writer's answer (no replica marker)" + ); + + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// [`replica_window_failure_falls_back_to_writer`] for the thread-replies + /// path: a replica-routed thread page whose query errors after the proof + /// re-runs on the writer instead of surfacing an error. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn replica_thread_failure_falls_back_to_writer() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "fbt_w").await; + let (replica, rname) = create_scratch_db(&admin, "fbt_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + let root = signed_event_at(&author, "root", base); + for pool in [&writer, &replica] { + insert_top_level(pool, community, channel, &root).await; + } + let replies: Vec = (1..=3) + .map(|i| signed_event_at(&author, &format!("r{i}"), base + 10 * i as u64)) + .collect(); + for pool in [&writer, &replica] { + for reply in &replies { + insert_thread_reply(pool, community, channel, &root, reply).await; + } + } + // Replica-only divergent reply between r2 and r3 marks replica serves. + let ghost = signed_event_at(&author, "replica-only-ghost", base + 25); + insert_thread_reply(&replica, community, channel, &root, &ghost).await; + + let db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + + let page1 = db + .get_thread_replies(cid, root.id.as_bytes(), Some(10), 2, None) + .await + .expect("head page"); + let cur = thread_cursor(page1.last().expect("page 1 non-empty")); + + // Healthy: the full page after r2 is the replica's [ghost]. + let healthy = db + .get_thread_replies(cid, root.id.as_bytes(), Some(10), 1, Some(&cur)) + .await + .expect("healthy replica page"); + assert_eq!( + healthy[0].stored_event.event.content, "replica-only-ghost", + "fixture must route the cursor page to the replica while healthy" + ); + + sqlx::query("DROP TABLE events CASCADE") + .execute(&replica) + .await + .expect("drop replica events"); + + let page = db + .get_thread_replies(cid, root.id.as_bytes(), Some(10), 1, Some(&cur)) + .await + .expect("replica failure must fall back to the writer, not error"); + assert_eq!( + page[0].stored_event.event.content, "r3", + "fallback page must be the writer's answer" + ); + + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// Mid-request degradation of the held session (Dawn, review of + /// 1b0aa0dfa): when the proved replica transaction dies between the page + /// and an aux follow-up (stand-in: `pg_terminate_backend` on the reader + /// connection, the same tx-fatal shape as a recovery-conflict cancel), + /// [`ReadSession::query_events`] must re-run the query on the writer and + /// permanently degrade the session instead of surfacing the error. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn read_session_degrades_to_writer_when_replica_connection_dies() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "deg_w").await; + let (replica, rname) = create_scratch_db(&admin, "deg_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + let m1 = signed_event_at(&author, "m1", base); + let m2 = signed_event_at(&author, "m2", base + 10); + for pool in [&writer, &replica] { + for ev in [&m1, &m2] { + insert_top_level(pool, community, channel, ev).await; + } + } + // Writer-only row proves the degraded aux ran on the writer. + let fresh = signed_event_at(&author, "fresh-writer-only", base + 20); + insert_top_level(&writer, community, channel, &fresh).await; + + let db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + + let head = db + .get_channel_window(cid, channel, 1, None, None) + .await + .expect("head window"); + let cursor = head.next_cursor.expect("has_more implies next_cursor"); + let (_window, mut session) = db + .get_channel_window_with_session(cid, channel, 10, Some(cursor), None) + .await + .expect("routed cursor window"); + assert!( + session.is_replica(), + "fixture must route this page to the replica" + ); + + // Kill the reader's backend out from under the held transaction. + sqlx::query( + "SELECT pg_terminate_backend(pid) FROM pg_stat_activity \ + WHERE datname = $1 AND pid <> pg_backend_pid()", + ) + .bind(&rname) + .execute(&admin) + .await + .expect("terminate replica backends"); + + let mut aux = EventQuery::for_community(cid); + aux.channel_id = Some(channel); + let rows = session + .query_events(&aux) + .await + .expect("session must degrade to the writer, not error"); + assert!( + rows.iter() + .any(|se| se.event.content == "fresh-writer-only"), + "degraded aux must be served by the writer" + ); + assert!( + !session.is_replica(), + "the session must be permanently degraded to the writer" + ); + + drop(session); + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// Snapshot continuity (Wren, review of 17ea2ff6a): the routed request + /// runs inside ONE `REPEATABLE READ, READ ONLY` transaction whose first + /// statement was the heartbeat observation — so a row committed on the + /// replica *after* the proof must be invisible to every follow-up + /// statement in the same request (page, participants, aux). This + /// distinguishes the transaction contract from mere connection reuse: + /// autocommit statements on the same backend advance their snapshot + /// per statement and WOULD see the mid-request row. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn routed_request_holds_one_snapshot_across_page_and_aux() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "snap_w").await; + let (replica, rname) = create_scratch_db(&admin, "snap_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + let m1 = signed_event_at(&author, "m1", base); + let m2 = signed_event_at(&author, "m2", base + 10); + for pool in [&writer, &replica] { + for ev in [&m1, &m2] { + insert_top_level(pool, community, channel, ev).await; + } + } + + let db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + + // Head page on the writer yields the cursor for a replica-routed page. + let head = db + .get_channel_window(cid, channel, 1, None, None) + .await + .expect("head window"); + let cursor = head.next_cursor.expect("has_more implies next_cursor"); + + // Route the cursor page to the replica and HOLD the session. + let (window, mut session) = db + .get_channel_window_with_session(cid, channel, 10, Some(cursor), None) + .await + .expect("routed cursor window"); + assert!( + session.is_replica(), + "fixture must route this page to the replica" + ); + assert_eq!(window.rows.len(), 1, "page after m2 is [m1]"); + + // Mid-request: a new event commits on the replica (stands in for + // replay advancing between the page and the aux closure). + let mid = signed_event_at(&author, "mid-request-commit", base + 5); + insert_top_level(&replica, community, channel, &mid).await; + + // A fresh autocommit statement on ANOTHER session sees it — the row + // is really there (control for the assertion below). + let mut control = EventQuery::for_community(cid); + control.channel_id = Some(channel); + let visible_elsewhere = event::query_events(&replica, &control) + .await + .expect("control query"); + assert!( + visible_elsewhere + .iter() + .any(|se| se.event.content == "mid-request-commit"), + "control: the mid-request row must be committed and visible to a new snapshot" + ); + + // The held request session must NOT see it: its snapshot was + // anchored by the heartbeat observation, before the commit. + let mut aux = EventQuery::for_community(cid); + aux.channel_id = Some(channel); + let in_request = session.query_events(&aux).await.expect("aux query"); + assert!( + !in_request + .iter() + .any(|se| se.event.content == "mid-request-commit"), + "request transaction must hold the proof-time snapshot; a \ + mid-request commit leaking in means the aux ran outside the \ + request transaction (autocommit connection reuse)" + ); + // Rows from the proof-time snapshot are still served. + assert!( + in_request.iter().any(|se| se.event.content == "m1"), + "proof-time rows must remain visible in the request snapshot" + ); + + drop(session); + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// Head gate (Predicate A): with the budget unset, a head fetch reads + /// the writer even over an open fence; with a budget set and a fresh + /// proved entry, the head page is served by the replica session + /// (bounded staleness accepted); with a budget the fence entry exceeds, + /// the head page falls back to the writer. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn head_fetch_routes_by_configured_budget() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "head_w").await; + let (replica, rname) = create_scratch_db(&admin, "head_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + let shared = signed_event_at(&author, "shared", base); + for pool in [&writer, &replica] { + insert_top_level(pool, community, channel, &shared).await; + } + // Divergent heads prove which pool served the fetch. + let fresh = signed_event_at(&author, "fresh-writer-only", base + 30); + insert_top_level(&writer, community, channel, &fresh).await; + let marker = signed_event_at(&author, "replica-only-marker", base + 20); + insert_top_level(&replica, community, channel, &marker).await; + + let mut db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + let head_contents = |w: &thread::ChannelWindow| -> Vec { + w.rows + .iter() + .map(|r| r.stored_event.event.content.clone()) + .collect() + }; + + // Budget unset (rollout default): head → writer, fence open or not. + let head = db + .get_channel_window(cid, channel, 2, None, None) + .await + .expect("head, gate off"); + assert_eq!( + head_contents(&head), + vec!["fresh-writer-only".to_string(), "shared".to_string()], + "head routing must default off" + ); + + // Budget set, entry fresh (just recorded): head → replica. + db.set_replica_read_max_age_for_tests(Some(std::time::Duration::from_secs(5))); + let head = db + .get_channel_window(cid, channel, 2, None, None) + .await + .expect("head, gate on"); + assert_eq!( + head_contents(&head), + vec!["replica-only-marker".to_string(), "shared".to_string()], + "a fresh proved entry within budget must serve the head from the replica" + ); + + // Entry older than the budget: head falls back to the writer. + db.fence().close(); + db.fence().force_open_for_tests_at( + chrono::Utc::now(), + std::time::Instant::now() - std::time::Duration::from_secs(10), + ); + let head = db + .get_channel_window(cid, channel, 2, None, None) + .await + .expect("head, entry too old"); + assert_eq!( + head_contents(&head), + vec!["fresh-writer-only".to_string(), "shared".to_string()], + "an over-budget entry must fail the head gate closed" + ); + + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// End-to-end deploy-default proof for the NEW routed seams: with the + /// budget unset, a covered-eligible query (channel-pinned + `until`) + /// through [`Db::query_events_routed`] is served by the WRITER — the + /// `for_query` gate keeps the covered arm dark (rev 5). With the budget + /// set and a fresh proved entry, the same query routes to the replica. + /// Divergent fixtures prove which pool served each read. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn query_events_routed_defaults_dark_and_routes_covered_when_enabled() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "qer_w").await; + let (replica, rname) = create_scratch_db(&admin, "qer_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + let shared = signed_event_at(&author, "shared", base); + for pool in [&writer, &replica] { + insert_top_level(pool, community, channel, &shared).await; + } + let writer_only = signed_event_at(&author, "writer-only", base + 10); + insert_top_level(&writer, community, channel, &writer_only).await; + let replica_only = signed_event_at(&author, "replica-only", base + 20); + insert_top_level(&replica, community, channel, &replica_only).await; + + let mut db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + + // Covered-eligible shape: channel-pinned with an `until` upper + // bound below the (now) fence wall. + let q = { + let mut q = EventQuery::for_community(cid); + q.channel_id = Some(channel); + q.until = chrono::DateTime::from_timestamp((base + 60) as i64, 0); + q + }; + let contents = |evs: &[StoredEvent]| -> std::collections::BTreeSet { + evs.iter().map(|e| e.event.content.clone()).collect() + }; + + // Deploy default: budget unset ⇒ writer, even though the shape is + // covered-eligible and the fence is open. + let rows = db + .query_events_routed("test_routed", &q) + .await + .expect("routed query, gate off"); + assert!( + contents(&rows).contains("writer-only"), + "budget unset must serve the writer" + ); + assert!( + !contents(&rows).contains("replica-only"), + "budget unset must not reach the replica via the covered arm" + ); + + // Budget set ⇒ the covered arm serves it from the replica. + db.set_replica_read_max_age_for_tests(Some(std::time::Duration::from_secs(5))); + let rows = db + .query_events_routed("test_routed", &q) + .await + .expect("routed query, gate on"); + assert!( + contents(&rows).contains("replica-only"), + "budget set + covered-eligible must route to the replica" + ); + assert!(!contents(&rows).contains("writer-only")); + + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// COUNT is bounded-only (rev 5 deletion-visibility rule): a + /// covered-eligible shape must NOT let a count take the covered arm. + /// With the budget unset the count reads the WRITER even with an open + /// fence; with the budget set and a fresh entry it reads the replica + /// under the bounded arm. Divergent row counts prove the serving pool. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn count_events_routed_is_bounded_only() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "cnt_w").await; + let (replica, rname) = create_scratch_db(&admin, "cnt_r").await; + + let author = nostr::Keys::generate(); + let community = Uuid::new_v4(); + let channel = Uuid::new_v4(); + seed_community_channel(&writer, community, channel, &author).await; + seed_community_channel(&replica, community, channel, &author).await; + + let base = 1_700_000_000u64; + // Writer: 2 rows. Replica: 1 row. + for (i, content) in ["a", "b"].iter().enumerate() { + let ev = signed_event_at(&author, content, base + i as u64); + insert_top_level(&writer, community, channel, &ev).await; + } + let ev = signed_event_at(&author, "c", base); + insert_top_level(&replica, community, channel, &ev).await; + + let mut db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + let cid = CommunityId::from_uuid(community); + + // Covered-eligible shape on purpose: pinned + until. A count must + // ignore that eligibility. + let q = { + let mut q = EventQuery::for_community(cid); + q.channel_id = Some(channel); + q.until = chrono::DateTime::from_timestamp((base + 60) as i64, 0); + q + }; + + // Budget unset ⇒ bounded arm disabled ⇒ writer. + let n = db + .count_events_routed("test_count", &q) + .await + .expect("count, gate off"); + assert_eq!(n, 2, "budget unset must count on the writer"); + + // Budget set + fresh entry ⇒ bounded arm ⇒ replica. + db.set_replica_read_max_age_for_tests(Some(std::time::Duration::from_secs(5))); + let n = db + .count_events_routed("test_count", &q) + .await + .expect("count, gate on"); + assert_eq!(n, 1, "budget set must count on the replica (bounded)"); + + // Entry older than the budget ⇒ bounded fails ⇒ writer. Covered + // would still hold here (upper <= wall) — proving count never + // consults it. + db.fence().close(); + db.fence().force_open_for_tests_at( + chrono::Utc::now(), + std::time::Instant::now() - std::time::Duration::from_secs(10), + ); + let n = db + .count_events_routed("test_count", &q) + .await + .expect("count, entry too old"); + assert_eq!( + n, 2, + "an over-budget entry must fail the count closed to the writer, \ + even when the covered arm would admit the shape" + ); + + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// Community separation across every routed seam, verified on + /// REPLICA-SERVED reads. + /// + /// The pre-existing feed/event scoping tests prove the shared SQL + /// builders confine rows to one community, but they exercise those + /// builders through the WRITER wrapper. `_on` variants are + /// executor-only refactors, so scoping *should* be identical — this + /// test refuses to take that on faith and re-proves it through the + /// routed executor, on a snapshot the replica actually served. + /// + /// Construction: two communities A and B exist in BOTH databases with + /// the same ids. The replica additionally holds a `replica-only` row in + /// each — divergent fixtures, so any row bearing that content proves + /// the replica (not the writer) served the read. Every assertion + /// requests A and demands B's rows never appear, including B's + /// `replica-only` row, which is the one a leaky predicate would surface. + /// The routed fallback must cost ONE reader acquire budget, even when the + /// Aurora capability cache is cold. + /// + /// Regression test for a stacked-budget bug found at `9fa3c9c0b`: the + /// capability probe used to `acquire()` from the pool itself and return + /// `false` *uncached* on `PoolTimedOut`, so the routed read then spent a + /// SECOND `READER_ACQUIRE_TIMEOUT` inside `begin`. Measured 302ms against + /// a ~150ms documented bound. Boot priming + /// ([`Db::spawn_read_pool_boot_ping`]) hid it only when the boot ping + /// SUCCEEDED — and a reader that is unavailable at boot is exactly the + /// case the bound is specified for, so the two failures are correlated. + /// + /// The fixture reproduces that state deliberately: a size-1 reader whose + /// sole connection is established and then HELD (so every further acquire + /// must time out), with `reader_aurora_identity` asserted cold. It routes + /// through `count_events_routed` rather than calling `proved_reader` + /// directly, because `buzz_db_route_decision` is emitted by `route_read` + /// — a direct call would prove the timing but never emit the label. + /// + /// Timing uses an upper bound of 2x the budget minus a margin: it must + /// fail for two stacked budgets (~300ms) while tolerating scheduler + /// jitter on one (~150ms). Asserting a lower bound too would pin the + /// budget's own value, which `reader_acquire_timeout_is_the_documented_budget` + /// already covers. + #[tokio::test(flavor = "current_thread")] + #[ignore = "requires Postgres"] + async fn routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (seed, wname) = create_scratch_db(&admin, "one_budget").await; + seed.close().await; + let base = admin_url().await; + let scratch_url = { + let idx = base.rfind('/').expect("db url has a path segment"); + format!("{}/{}", &base[..idx], wname) + }; + + // `Db::new` so the writer arms the floor guard and the reader is the + // real lazy `connect_read_pool` pool (min_connections=0, 150ms + // acquire timeout). Reader is sized 1 so holding one connection + // saturates it. + let mut db = Db::new(&DbConfig { + database_url: scratch_url.clone(), + read_database_url: Some(scratch_url), + max_connections: 4, + read_max_connections: Some(1), + ..DbConfig::default() + }) + .await + .expect("connect armed Db with size-1 lazy reader"); + db.fence().force_open_for_tests(chrono::Utc::now()); + db.set_replica_read_max_age_for_tests(Some(Duration::from_secs(5))); + + let read_pool = db.read_pool.clone().expect("reader pool configured"); + // Establish and hold the reader's only connection: saturated. + let held = read_pool + .acquire() + .await + .expect("establish the reader's sole connection"); + assert_eq!( + db.read_max_connections, 1, + "reader max must report 1 for this fixture to test saturation" + ); + assert_eq!( + read_pool.size(), + 1, + "the sole reader connection is established and held" + ); + // The bug is only observable with the capability cache cold; if a + // future change primes it here, this fixture would silently stop + // discriminating. + assert!( + db.reader_aurora_identity.get().is_none(), + "Aurora capability must be UNPRIMED (post-boot-ping-failure state)" + ); + + let recorder = metrics_util::debugging::DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + let query = EventQuery::for_community(CommunityId::from_uuid(Uuid::new_v4())); + + // The recorder is installed thread-locally, so it must stay installed + // across the `.await` — hence the guard form rather than + // `with_local_recorder`, whose closure cannot host an await. The + // `current_thread` flavor keeps the route decision on this thread; on + // a multi-thread runtime the emit could land on a worker where no + // local recorder is installed and the label assertions would vacuously + // see an empty snapshot. + let start = std::time::Instant::now(); + let count = { + let _guard = metrics::set_default_local_recorder(&recorder); + db.count_events_routed("one_budget_probe", &query).await + } + .expect("writer fallback still answers the read"); + let elapsed = start.elapsed(); + + assert_eq!(count, 0, "writer answered on an empty scratch database"); + assert!( + elapsed < Duration::from_millis(250), + "routed fallback must spend ONE {}ms acquire budget, not two; took {}ms", + Db::READER_ACQUIRE_TIMEOUT.as_millis(), + elapsed.as_millis() + ); + + let reasons: std::collections::HashMap<(String, String), u64> = snapshotter + .snapshot() + .into_vec() + .into_iter() + .filter(|(key, ..)| key.key().name() == "buzz_db_route_decision") + .map(|(key, _, _, value)| { + let metrics_util::debugging::DebugValue::Counter(n) = value else { + panic!("buzz_db_route_decision must be a counter"); + }; + let labels: Vec<_> = key.key().labels().collect(); + let get = |name: &str| { + labels + .iter() + .find(|l| l.key() == name) + .map(|l| l.value().to_owned()) + .unwrap_or_default() + }; + ((get("decision"), get("reason")), n) + }) + .collect(); + + assert_eq!( + reasons.get(&("writer".to_owned(), "reader_acquire_timeout".to_owned())), + Some(&1), + "saturated reader must fall back as writer/reader_acquire_timeout; got {reasons:?}" + ); + // `reader_validation_error` would mean we misclassified a timeout as a + // broken reader, and `pool_busy` is the retired name — neither may + // appear in ANY emitted label. + assert!( + !reasons + .keys() + .any(|(_, reason)| reason == "reader_validation_error" || reason == "pool_busy"), + "no reader_validation_error or retired pool_busy label may be emitted; got {reasons:?}" + ); + + drop(held); + drop_scratch_db(&admin, db.pool.clone(), &wname).await; + } + + #[tokio::test] + #[ignore = "requires Postgres"] + async fn routed_reads_are_confined_to_the_requested_community() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (writer, wname) = create_scratch_db(&admin, "sep_w").await; + let (replica, rname) = create_scratch_db(&admin, "sep_r").await; + + let author = nostr::Keys::generate(); + let (comm_a, chan_a) = (Uuid::new_v4(), Uuid::new_v4()); + let (comm_b, chan_b) = (Uuid::new_v4(), Uuid::new_v4()); + for pool in [&writer, &replica] { + seed_community_channel(pool, comm_a, chan_a, &author).await; + seed_community_channel(pool, comm_b, chan_b, &author).await; + } + + // A p-tag mention is what makes a row eligible for the mentions and + // needs-action feeds. Kind 9 satisfies mentions + activity; + // needs-action admits only approval/reminder kinds, so each + // community also gets a kind-46010 row. + let mentioned = nostr::Keys::generate(); + let mentioned_hex = mentioned.public_key().to_hex(); + let mentioned_bytes = mentioned.public_key().to_bytes(); + let tagged_kind = |kind: u16, content: &str, secs: u64| { + nostr::EventBuilder::new(nostr::Kind::Custom(kind), content) + .tags([nostr::Tag::parse(["p", mentioned_hex.as_str()]).expect("p tag")]) + .custom_created_at(nostr::Timestamp::from(secs)) + .sign_with_keys(&author) + .expect("sign event") + }; + let tagged = |content: &str, secs: u64| tagged_kind(9, content, secs); + + let base = 1_700_000_000u64; + // Shared rows (both DBs) + replica-only rows (divergence) per community. + let a_shared = tagged("a-shared", base); + let b_shared = tagged("b-shared", base + 1); + for pool in [&writer, &replica] { + insert_top_level(pool, comm_a, chan_a, &a_shared).await; + insert_mentions( + pool, + CommunityId::from_uuid(comm_a), + &a_shared, + Some(chan_a), + ) + .await + .expect("mentions a-shared"); + insert_top_level(pool, comm_b, chan_b, &b_shared).await; + insert_mentions( + pool, + CommunityId::from_uuid(comm_b), + &b_shared, + Some(chan_b), + ) + .await + .expect("mentions b-shared"); + } + let a_replica_only = tagged("a-replica-only", base + 10); + let b_replica_only = tagged("b-replica-only", base + 11); + insert_top_level(&replica, comm_a, chan_a, &a_replica_only).await; + insert_mentions( + &replica, + CommunityId::from_uuid(comm_a), + &a_replica_only, + Some(chan_a), + ) + .await + .expect("mentions a-replica-only"); + insert_top_level(&replica, comm_b, chan_b, &b_replica_only).await; + insert_mentions( + &replica, + CommunityId::from_uuid(comm_b), + &b_replica_only, + Some(chan_b), + ) + .await + .expect("mentions b-replica-only"); + + // Needs-action fixtures: approval kind, replica-only in BOTH + // communities, so the assertion below is replica-served on A and + // must still not see B's. + let a_approval = tagged_kind(46010, "a-approval-replica-only", base + 20); + let b_approval = tagged_kind(46010, "b-approval-replica-only", base + 21); + insert_top_level(&replica, comm_a, chan_a, &a_approval).await; + insert_mentions( + &replica, + CommunityId::from_uuid(comm_a), + &a_approval, + Some(chan_a), + ) + .await + .expect("mentions a-approval"); + insert_top_level(&replica, comm_b, chan_b, &b_approval).await; + insert_mentions( + &replica, + CommunityId::from_uuid(comm_b), + &b_approval, + Some(chan_b), + ) + .await + .expect("mentions b-approval"); + + let mut db = Db::from_pools(writer.clone(), replica.clone()); + db.fence().force_open_for_tests(chrono::Utc::now()); + db.set_replica_read_max_age_for_tests(Some(std::time::Duration::from_secs(5))); + let cid_a = CommunityId::from_uuid(comm_a); + + let contents = |evs: &[StoredEvent]| -> std::collections::BTreeSet { + evs.iter().map(|e| e.event.content.clone()).collect() + }; + // Every routed seam must (a) have been served by the replica — + // proven by a divergent row absent from the writer — and (b) contain + // no row belonging to community B. All B fixtures are named `b-*`, + // so the leak check is a single prefix scan. + let assert_a_only = |rows: &[StoredEvent], marker: &str, seam: &str| { + let got = contents(rows); + assert!( + got.contains(marker), + "{seam}: must be replica-served (divergent row `{marker}` absent from writer); got {got:?}" + ); + assert!( + !got.iter().any(|c| c.starts_with("b-")), + "{seam}: community B rows leaked into a community A read; got {got:?}" + ); + }; + + // 1. Generic query — covered arm (channel-pinned + `until`). + let mut q = EventQuery::for_community(cid_a); + q.channel_id = Some(chan_a); + q.until = chrono::DateTime::from_timestamp((base + 60) as i64, 0); + let rows = db + .query_events_routed("sep_query", &q) + .await + .expect("routed query"); + assert_a_only(&rows, "a-replica-only", "query_events_routed"); + + // 2. Generic query — bounded arm (no channel pin at all, so a + // missing community predicate could not be masked by the pin). + let unpinned = EventQuery::for_community(cid_a); + let rows = db + .query_events_routed_bounded("sep_query_bounded", &unpinned) + .await + .expect("routed bounded query"); + assert_a_only(&rows, "a-replica-only", "query_events_routed_bounded"); + + // 3. COUNT — bounded-only. Community A holds 3 rows on the replica + // (shared + replica-only + approval) but only 1 on the writer, + // and 3 more exist in community B. Exactly 3 proves the read was + // both replica-served and community-confined. + let count = db + .count_events_routed("sep_count", &unpinned) + .await + .expect("routed count"); + assert_eq!( + count, 3, + "count must see A's three replica rows only — not B's, not the writer's one" + ); + + // 4. By-ID hydration — ids carry no channel pin, and B's ids are + // requested alongside A's. Only A's may hydrate. + let ids: Vec<&[u8]> = vec![ + a_shared.id.as_bytes(), + a_replica_only.id.as_bytes(), + b_shared.id.as_bytes(), + b_replica_only.id.as_bytes(), + ]; + let rows = db + .get_events_by_ids_routed("sep_by_ids", cid_a, &ids) + .await + .expect("routed by-ids"); + assert_a_only(&rows, "a-replica-only", "get_events_by_ids_routed"); + + // 5-7. All three feed builders, each given BOTH channels as + // accessible — so only the community predicate can exclude B. + let both = [chan_a, chan_b]; + let rows = db + .query_feed_mentions_routed("sep_feed", cid_a, &mentioned_bytes, &both, None, 50) + .await + .expect("routed mentions"); + assert_a_only(&rows, "a-replica-only", "query_feed_mentions_routed"); + + let rows = db + .query_feed_needs_action_routed("sep_feed", cid_a, &mentioned_bytes, &both, None, 50) + .await + .expect("routed needs action"); + assert_a_only( + &rows, + "a-approval-replica-only", + "query_feed_needs_action_routed", + ); + + let rows = db + .query_feed_activity_routed("sep_feed", cid_a, &both, None, 50) + .await + .expect("routed activity"); + assert_a_only(&rows, "a-replica-only", "query_feed_activity_routed"); + + drop_scratch_db(&admin, replica, &rname).await; + drop_scratch_db(&admin, writer, &wname).await; + } + + /// D4: a LAZY reader pool (connect_lazy, min_connections=0, never yet + /// used) must still let [`Db::spawn_fence_probe`] verify the writer's + /// floor guard and spawn — reader-down or reader-idle at boot must not + /// disable fence probing. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn lazy_reader_pool_still_spawns_fence_probe() { + let admin = PgPool::connect(&admin_url().await) + .await + .expect("connect admin"); + let (seed, wname) = create_scratch_db(&admin, "lazy_w").await; + seed.close().await; + + let writer_url = { + let base = admin_url().await; + let idx = base.rfind('/').expect("db url has a path segment"); + format!("{}/{}", &base[..idx], wname) + }; + // `Db::new` (not `from_pools`) so the WRITER pool arms the + // `buzz.created_at_floor` GUC — `spawn_fence_probe` verifies the + // floor guard on a writer connection, and `create_scratch_db`'s + // plain `PgPool::connect` never arms it. The reader is still the + // lazy `connect_read_pool` pool this test is about. + let db = Db::new(&DbConfig { + database_url: writer_url.clone(), + read_database_url: Some(writer_url), + max_connections: 2, + ..DbConfig::default() + }) + .await + .expect("connect armed Db with lazy reader"); + + let spawned = db + .spawn_fence_probe() + .await + .expect("floor-guard verification must pass on the migrated writer"); + assert!(spawned, "a configured (lazy) reader must spawn the probe"); + + drop_scratch_db(&admin, db.pool.clone(), &wname).await; + } + /// Thread replies: head fetch reads the writer; a FULL cursor page is /// served by the replica; an UNDER-limit cursor page (candidate terminal /// page) is re-run on the writer so a lagged replica can never truncate @@ -6012,21 +8212,39 @@ mod tests { let base = admin_url().await; let idx = base.rfind('/').expect("db url has a path segment"); - let db = Db::new(&DbConfig { - database_url: format!("{}/{}", &base[..idx], wname), - read_database_url: Some(format!("{}/{}", &base[..idx], rname)), + let writer_url = format!("{}/{}", &base[..idx], wname); + let replica_url = format!("{}/{}", &base[..idx], rname); + + // Healthy schema: verification passes, probe starts. A SEPARATE Db + // instance, because its background probe legitimately opens its own + // fence (the heartbeat probe is writer-side only) — the refusal + // assertions below must run against a fence whose spawns were all + // refused. + let db_healthy = Db::new(&DbConfig { + database_url: writer_url.clone(), + read_database_url: Some(replica_url.clone()), max_connections: 2, ..DbConfig::default() }) .await .expect("connect armed Db with replica"); - - // Healthy schema: verification passes, probe starts. assert!( - db.spawn_fence_probe().await.expect("verification passes"), + db_healthy + .spawn_fence_probe() + .await + .expect("verification passes"), "probe must start on a verified schema" ); + let db = Db::new(&DbConfig { + database_url: writer_url, + read_database_url: Some(replica_url), + max_connections: 2, + ..DbConfig::default() + }) + .await + .expect("connect armed Db with replica"); + // Sabotage A: catalog-shaped no-op — same trigger, gutted function // body. Catalog check alone would pass; behavior check must refuse. sqlx::query( @@ -6066,6 +8284,10 @@ mod tests { "fence must remain closed when verification refuses the probe" ); + db_healthy.pool.close().await; + if let Some(rp) = &db_healthy.read_pool { + rp.close().await; + } db.pool.close().await; if let Some(rp) = &db.read_pool { rp.close().await; diff --git a/crates/buzz-db/src/migration.rs b/crates/buzz-db/src/migration.rs index 1d1b7e05d4..6985916bba 100644 --- a/crates/buzz-db/src/migration.rs +++ b/crates/buzz-db/src/migration.rs @@ -347,6 +347,7 @@ mod tests { "push_gateway_delivery_auth_replays", "push_gateway_delivery_request_replays", "product_feedback", + "replica_heartbeat", ] { if normalized[insert_pos..].contains(&format!("'{value}'")) { globals.insert(value.to_owned()); @@ -560,7 +561,7 @@ mod tests { let mut migrations: Vec<_> = MIGRATOR.iter().collect(); migrations.sort_by_key(|migration| migration.version); - assert_eq!(migrations.len(), 25); + assert_eq!(migrations.len(), 26); assert_eq!(migrations[0].version, 1); assert_eq!(&*migrations[0].description, "initial schema"); assert!(migrations[0] @@ -904,6 +905,20 @@ mod tests { desired_schema.contains("CREATE TABLE join_policy_acceptances"), "desired-state schema must include join-policy evidence used by invite claims", ); + + // Replica heartbeat (this branch, renumbered to 0026 after + // 0025_relay_invites landed on main): the fence's portable read-side + // observation. A single CHECK'd row makes the token update the + // serialization point (multi-pod commit ordering), and the epoch + // column is what detects token resets — both are load-bearing for + // the routing proof. + assert_eq!(migrations[25].version, 26); + let heartbeat = migrations[25].sql.as_str(); + assert!(heartbeat.contains("CREATE TABLE replica_heartbeat")); + assert!(heartbeat.contains("CHECK (id = 1)")); + assert!(heartbeat.contains("epoch")); + assert!(heartbeat.contains("INSERT INTO replica_heartbeat (id) VALUES (1)")); + assert!(heartbeat.contains("_operator_global_tables")); } #[test] @@ -1146,7 +1161,7 @@ mod tests { run_migrations(&pool) .await .expect("retry succeeds after operator repair"); - assert_eq!(applied_versions(&pool).await.last().copied(), Some(25)); + assert_eq!(applied_versions(&pool).await.last().copied(), Some(26)); } #[tokio::test] diff --git a/crates/buzz-db/src/replica_fence.rs b/crates/buzz-db/src/replica_fence.rs index 03bc1c77f0..c840db393e 100644 --- a/crates/buzz-db/src/replica_fence.rs +++ b/crates/buzz-db/src/replica_fence.rs @@ -9,38 +9,59 @@ //! (`clock_timestamp()`, evaluated inside commit processing). Enforcement //! is armed per session via the `buzz.created_at_floor` GUC, which the //! relay's writer pool sets on every connection. -//! 2. **Ordered LSN handshake** (this module): on one pinned writer -//! connection, three separately-awaited statements sample +//! 2. **Ordered heartbeat handshake** (this module): on one pinned writer +//! connection, separately-awaited statements sample //! `S = clock_timestamp()`, then scan `pg_stat_activity` for the oldest -//! open transaction, then capture `L = pg_current_wal_lsn()` **last**. -//! Once the replica reports `pg_last_wal_replay_lsn() >= L`, every -//! transaction partitions into exactly three buckets: -//! (a) finished before the activity scan — its commit WAL precedes `L`, -//! so the replica has replayed it; +//! open transaction, then — **last** — commit heartbeat token `M` via a +//! single-row `UPDATE replica_heartbeat ... RETURNING token, epoch` +//! (migration 0026). Because the single-row UPDATE serializes all pods' +//! probes, tokens are globally commit-ordered. A reader **session** that +//! observes `token >= M` on its own connection has, by WAL/storage replay +//! order, also replayed every commit that preceded M's commit; every +//! transaction then partitions into exactly three buckets: +//! (a) finished before the activity scan — its commit precedes `M`'s +//! commit, so the replica session has replayed it; //! (b) open at the activity scan — represented by `xact_start`, so it is //! bounded by the `oldest_xact_start` term; //! (c) started after the activity scan — its deferred floor guard runs //! after `S`, so it cannot commit a row with //! `created_at < S - floor`. -//! There is no fourth bucket. The fence therefore advances to -//! `min(oldest_xact_start, S) - floor - clock_margin`, and every -//! channel-window row with `created_at <= fence` is on the replica. +//! There is no fourth bucket. Each committed token `M` therefore proves a +//! **fence wall** of `min(oldest_xact_start, S) - floor - clock_margin`: +//! every channel-window row with `created_at <= fence_wall(M)` is present +//! on any reader session observing `token >= M`. +//! +//! Unlike the previous WAL-LSN observation (`pg_last_wal_replay_lsn()`, which +//! Aurora reader endpoints hide), the token observation is portable and — +//! critically — **snapshot-local**: routing opens a `REPEATABLE READ, READ +//! ONLY` transaction on the reader session that will serve the page and +//! observes the heartbeat as its first statement, so the proof binds to the +//! exact snapshot every follow-up statement in the request (page, +//! participants, aux closure) reads from — never to a different pooled +//! session (readers behind one endpoint may sit at different replay +//! positions), and never to a later autocommit snapshot on the same wire. +//! An observed token lower than the newest retained `M` is ordinary +//! replication lag, not a fault; the resolver simply proves from an older +//! retained `M`. Regression detection is writer-side only: a non-monotonic +//! `RETURNING token` or an epoch change (restore/re-seed) clears the retained +//! ring, so no stale entry can masquerade as fresh coverage. //! //! Everything fails **closed**: probe errors, masked `pg_stat_activity` -//! visibility, NULL/absent replica LSN (Aurora observability differences), -//! non-advancing replay, or probe staleness all close the fence, which routes -//! all reads back to the writer — degraded capacity, never holes. +//! visibility, an unreadable heartbeat row on the reader session, an epoch +//! mismatch, or an observed token below every retained entry all route the +//! request back to the writer — degraded capacity, never holes. //! //! Operational bypasses (sessions without the GUC, `session_replication_role //! = replica` restores) are outside the proof by design and require holding //! the fence closed for their duration; see `migrations/0021`. -use std::sync::atomic::{AtomicI64, Ordering}; -use std::sync::Arc; -use std::time::Duration; +use std::collections::VecDeque; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; use chrono::{DateTime, Utc}; -use sqlx::{PgPool, Row}; +use sqlx::{PgConnection, PgPool, Row}; +use uuid::Uuid; /// Seconds of `created_at` history the commit-time floor guard tolerates. /// @@ -58,79 +79,231 @@ pub const CREATED_AT_FLOOR_SECS: i64 = 960; /// between machines. pub const FENCE_CLOCK_MARGIN_SECS: i64 = 5; -/// How often the probe samples the writer and checks the replica. -pub const PROBE_INTERVAL: Duration = Duration::from_secs(5); +/// How often the probe samples the writer and commits a heartbeat token. +/// +/// 500ms keeps the cadence at least 2x under the smallest sensible bounded +/// budget (`BUZZ_REPLICA_READ_MAX_AGE_MS`, deploy plan 1000ms) so +/// eligibility doesn't flap between beats. Cost is one single-row UPDATE +/// tuple of WAL per beat per pod — ~20 beats/s fleet-wide, <0.1% of the +/// writer. +pub const PROBE_INTERVAL: Duration = Duration::from_millis(500); -/// A fence older than this is stale: the probe has stopped confirming -/// freshness and the fence closes until a new handshake completes. +/// A fence whose newest entry is older than this is stale: the probe has +/// stopped committing tokens and routing eligibility closes until a new +/// handshake completes. +/// +/// Note this is an availability hygiene gate, not a soundness requirement: +/// a retained entry's proof (`token >= M` on a session implies every row +/// `<= fence_wall(M)` is present there) never decays. Closing on staleness +/// just stops spending reader checkouts once the probe is evidently dead. pub const FENCE_STALENESS: Duration = Duration::from_secs(30); -/// Sentinel: fence closed (no verified replica coverage). -const CLOSED: i64 = i64::MIN; +/// How many `(token, fence_wall)` entries the fence retains. At one probe +/// per [`PROBE_INTERVAL`] (500ms) this is ~60 seconds of history — a reader +/// session lagging further than that behind the newest token fails closed +/// (routes to the writer) rather than proving from thin air. Aurora reader +/// lag is typically tens of milliseconds; a reader minutes behind is a +/// fault, not a routing candidate. +const RING_CAPACITY: usize = 120; -/// Shared fence state. `Db` holds an `Arc` of this; the probe task advances -/// it and cursor routing consults it. -#[derive(Debug)] +// The retained window must outlast the staleness gate: if the ring held +// less than FENCE_STALENESS of history, a non-stale newest entry could +// coexist with proved-but-evicted older entries, failing sessions closed +// for capacity rather than lag. Compile-checked so a future cadence or +// capacity tweak can't silently shrink the window below the gate. +const _: () = assert!( + RING_CAPACITY as u64 * PROBE_INTERVAL.as_millis() as u64 > FENCE_STALENESS.as_millis() as u64, + "fence ring must retain more history than the staleness gate" +); + +/// One retained heartbeat observation: proof material for reader sessions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TokenEntry { + /// The committed heartbeat token `M`. + pub token: i64, + /// Monotonic instant captured just before `M` was committed. `elapsed()` + /// bounds (from above) how old a session observing `token >= M` can be — + /// the freshness term of the head-routing predicate. + pub committed_at: Instant, + /// `min(oldest_xact_start, S) - floor - clock_margin` for `M`'s + /// handshake: every channel-window row with `created_at <= fence_wall` + /// is present on any session observing `token >= M`. + pub fence_wall: DateTime, +} + +#[derive(Debug, Default)] +struct FenceInner { + /// Epoch the retained ring belongs to. `None` until the first probe — + /// or after the test hook, whose injected entry deliberately bypasses + /// the epoch comparison in [`ReplicaFence::resolve`]. + epoch: Option, + /// Retained entries in strictly increasing token order. + ring: VecDeque, +} + +/// Outcome of recording one probe sample. +#[derive(Debug, PartialEq, Eq)] +pub enum RecordOutcome { + /// Entry retained; proofs may cite it. + Recorded, + /// The token went backwards within the same epoch — a restore that kept + /// the old epoch. The ring was cleared and the entry discarded; the + /// probe must rotate the epoch before recording again (a reader still on + /// the pre-rewind timeline could otherwise observe a *higher* token that + /// proves nothing about the new timeline). + TokenRegression, +} + +/// Outcome of resolving one reader-session observation against the ring. +/// Everything but [`ResolveOutcome::Proved`] fails closed (routes to the +/// writer); the variants exist so route metrics can name the reason. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ResolveOutcome { + /// The observation proves this retained entry. + Proved(TokenEntry), + /// The observed epoch is not the ring's epoch — the session is on a + /// different timeline (restore) or the ring was rotated under it. + EpochMismatch, + /// The observed token is below every retained entry: the reader lags + /// further than the ring's history (or the ring is empty). + TokenBehind, +} + +impl ResolveOutcome { + /// The proved entry, if any. + pub fn proved(self) -> Option { + match self { + ResolveOutcome::Proved(entry) => Some(entry), + _ => None, + } + } +} + +/// Shared fence state. `Db` holds an `Arc` of this; the probe task records +/// entries and per-request routing resolves proofs against it. +#[derive(Debug, Default)] pub struct ReplicaFence { - /// Unix micros of the newest verified-complete timestamp, or `CLOSED`. - fence_micros: AtomicI64, - /// Unix micros when the fence was last advanced (staleness check). - updated_micros: AtomicI64, + inner: Mutex, } impl ReplicaFence { - /// A new fence, initially closed. + /// A new fence, initially closed (empty ring). pub fn new() -> Self { - Self { - fence_micros: AtomicI64::new(CLOSED), - updated_micros: AtomicI64::new(CLOSED), - } + Self::default() } - /// Close the fence: all cursor reads route to the writer. + /// Close the fence: drop all retained proofs; reads route to the writer. pub fn close(&self) { - self.fence_micros.store(CLOSED, Ordering::Relaxed); + let mut inner = self.inner.lock().expect("fence lock poisoned"); + inner.ring.clear(); } - fn advance(&self, fence: DateTime) { - self.fence_micros - .store(fence.timestamp_micros(), Ordering::Relaxed); - self.updated_micros - .store(Utc::now().timestamp_micros(), Ordering::Relaxed); + /// Record one probe sample. Epoch changes (re-seed) clear the ring and + /// start a new one under the observed epoch — sound, because an entry + /// only proves commits on its own timeline and readers must match the + /// epoch to cite it. A same-epoch token regression is the unsafe case: + /// see [`RecordOutcome::TokenRegression`]. + pub fn record( + &self, + token: i64, + epoch: Uuid, + committed_at: Instant, + fence_wall: DateTime, + ) -> RecordOutcome { + let mut inner = self.inner.lock().expect("fence lock poisoned"); + if inner.epoch != Some(epoch) { + inner.ring.clear(); + inner.epoch = Some(epoch); + } else if inner.ring.back().is_some_and(|last| token <= last.token) { + inner.ring.clear(); + return RecordOutcome::TokenRegression; + } + if inner.ring.len() == RING_CAPACITY { + inner.ring.pop_front(); + } + inner.ring.push_back(TokenEntry { + token, + committed_at, + fence_wall, + }); + RecordOutcome::Recorded } - /// The current fence, or `None` when closed or stale. + /// Resolve the strongest proof a reader session's observation supports: + /// the greatest retained entry with `entry.token <= observed_token`, + /// provided the observed epoch matches the ring's. Non-`Proved` outcomes + /// fail closed; they are distinguished only for route metrics. + pub fn resolve(&self, observed_token: i64, observed_epoch: Uuid) -> ResolveOutcome { + let inner = self.inner.lock().expect("fence lock poisoned"); + match inner.epoch { + Some(e) if e != observed_epoch => return ResolveOutcome::EpochMismatch, + // `None` with a non-empty ring only happens via the test hook; + // the epoch comparison is deliberately skipped there. + _ => {} + } + inner + .ring + .iter() + .rev() + .find(|entry| entry.token <= observed_token) + .copied() + .map_or(ResolveOutcome::TokenBehind, ResolveOutcome::Proved) + } + + /// The newest retained entry, staleness-gated. Used as the cheap + /// pre-check before spending a reader checkout, and for observability. + pub fn newest(&self) -> Option { + let inner = self.inner.lock().expect("fence lock poisoned"); + inner + .ring + .back() + .filter(|entry| entry.committed_at.elapsed() <= FENCE_STALENESS) + .copied() + } + + /// Age of the newest retained entry, ungated (observability: how long + /// since the probe last committed a token). + pub fn heartbeat_age(&self) -> Option { + let inner = self.inner.lock().expect("fence lock poisoned"); + inner.ring.back().map(|entry| entry.committed_at.elapsed()) + } + + /// The newest fence wall, or `None` when closed or stale. /// - /// Rows with `created_at <= fence` are verified present on the replica. + /// Rows with `created_at <= fence` are verified present on a reader + /// session that proves the newest entry; whether a *given* session does + /// is decided per request via [`ReplicaFence::resolve`]. pub fn verified_through(&self) -> Option> { - let raw = self.fence_micros.load(Ordering::Relaxed); - if raw == CLOSED { - return None; - } - let updated = self.updated_micros.load(Ordering::Relaxed); - let age_micros = Utc::now().timestamp_micros().saturating_sub(updated); - if age_micros > FENCE_STALENESS.as_micros() as i64 { - return None; - } - DateTime::from_timestamp_micros(raw) + self.newest().map(|entry| entry.fence_wall) } - /// Whether the replica verifiably holds every channel-window row at or - /// before `ts`. + /// Whether some retained entry's wall covers `ts` — the cheap routing + /// pre-check (the connection-local observation still has to prove it). pub fn covers(&self, ts: DateTime) -> bool { self.verified_through().is_some_and(|fence| ts <= fence) } /// Test hook: force the fence open through `ts` without a probe. - /// Used by routing tests that stand up a divergent fake replica. + /// Injects an entry any observed token satisfies (`i64::MIN`) with no + /// epoch recorded, so the epoch comparison is bypassed — routing tests + /// stand up a divergent fake replica whose heartbeat epoch differs from + /// the writer's. pub fn force_open_for_tests(&self, ts: DateTime) { - self.advance(ts); + self.force_open_for_tests_at(ts, Instant::now()); } -} -impl Default for ReplicaFence { - fn default() -> Self { - Self::new() + /// [`ReplicaFence::force_open_for_tests`] with an explicit commit + /// instant, for pinning age-gated behavior (head-budget and staleness + /// tests inject entries "committed" in the past). + pub fn force_open_for_tests_at(&self, ts: DateTime, committed_at: Instant) { + let mut inner = self.inner.lock().expect("fence lock poisoned"); + inner.epoch = None; + inner.ring.clear(); + inner.ring.push_back(TokenEntry { + token: i64::MIN, + committed_at, + fence_wall: ts, + }); } } @@ -332,15 +505,20 @@ struct WriterSample { /// Oldest open transaction among other client backends at scan time, /// or `None` when no transaction was open. oldest_xact_start: Option>, - /// `L`: writer `pg_current_wal_lsn()` captured last, as text. - wal_lsn: String, + /// `M`: the heartbeat token committed **last**, after the scan. + token: i64, + /// Heartbeat epoch returned with `M`. + epoch: Uuid, + /// Monotonic instant captured immediately before committing `M` — an + /// upper bound on how old a session observing `token >= M` can be. + committed_at: Instant, } /// Errors that close the fence. All variants are logged and treated /// identically: fail closed. #[derive(Debug, thiserror::Error)] pub enum ProbeError { - /// A probe query against writer or replica failed. + /// A probe query against the writer failed. #[error("writer probe query failed: {0}")] Writer(#[from] sqlx::Error), /// `pg_stat_activity` hid state for another backend that could hold an @@ -353,14 +531,15 @@ pub enum ProbeError { /// Number of other client backends with masked/unknown state. masked: i64, }, - /// The replica returned NULL for the replay-LSN comparison. - #[error("replica did not report a comparable replay LSN")] - ReplicaLsnUnavailable, + /// The single heartbeat row (migration 0026) is missing on the writer. + #[error("replica_heartbeat row missing on the writer — migration 0026 not applied?")] + HeartbeatRowMissing, } -/// Take one ordered writer sample: S, then activity scan, then L **last**. +/// Take one ordered writer sample: S, then activity scan, then commit the +/// heartbeat token **last**. /// -/// The three statements are separately awaited on a single pinned connection; +/// The statements are separately awaited on a single pinned connection; /// a single SELECT would not guarantee evaluation order across the /// subexpressions, reopening the race this ordering exists to close. async fn sample_writer(writer: &PgPool) -> Result { @@ -393,8 +572,8 @@ async fn sample_writer(writer: &PgPool) -> Result { // // Prepared transactions (2PC) are a bucket of their own: while // prepared they have left `pg_stat_activity` but can still commit - // after `L`. Their deferred floor guard already ran at PREPARE, so - // `pg_prepared_xacts.prepared` bounds their rows exactly like + // after the token. Their deferred floor guard already ran at PREPARE, + // so `pg_prepared_xacts.prepared` bounds their rows exactly like // `xact_start`; fold it into the same minimum. let row = sqlx::query( r#" @@ -423,80 +602,171 @@ async fn sample_writer(writer: &PgPool) -> Result { } let oldest_xact_start: Option> = row.get("oldest_xact_start"); - // 3. L last. - let wal_lsn: String = sqlx::query_scalar("SELECT pg_current_wal_lsn()::text") - .fetch_one(&mut *conn) - .await?; + // 3. Token commit LAST, on the same pinned connection. The single-row + // UPDATE serializes concurrent pods' probes, so RETURNING token is + // globally commit-ordered. `committed_at` is captured before the + // round trip so `elapsed()` over-estimates the observation's age — + // the conservative direction for the head-freshness bound. + let committed_at = Instant::now(); + let row = sqlx::query( + "UPDATE replica_heartbeat SET token = token + 1 WHERE id = 1 RETURNING token, epoch", + ) + .fetch_optional(&mut *conn) + .await? + .ok_or(ProbeError::HeartbeatRowMissing)?; Ok(WriterSample { sampled_at, oldest_xact_start, - wal_lsn, + token: row.get("token"), + epoch: row.get("epoch"), + committed_at, }) } -/// Whether the replica has replayed at least through `wal_lsn`. -/// -/// The comparison happens on the replica in pg_lsn domain. The -/// `pg_is_in_recovery()` gate is load-bearing: after crash recovery or -/// promotion a *primary* returns a static non-NULL `pg_last_wal_replay_lsn()` -/// rather than NULL, so NULL-checking alone would not reliably detect a -/// misrouted "replica" URL. Not-in-recovery, NULL replay LSN, or Aurora -/// hiding either is an error → fence closes. -async fn replica_covers(replica: &PgPool, wal_lsn: &str) -> Result { - let covered: Option = sqlx::query_scalar( - r#" - SELECT CASE - WHEN pg_is_in_recovery() THEN pg_last_wal_replay_lsn() >= $1::pg_lsn - ELSE NULL - END - "#, - ) - .bind(wal_lsn) - .fetch_one(replica) - .await?; - covered.ok_or(ProbeError::ReplicaLsnUnavailable) +/// The fence wall proved by one handshake: +/// `min(oldest_xact_start, S) - floor - clock_margin`. +fn fence_wall(sample_s: DateTime, oldest_xact_start: Option>) -> DateTime { + let lower = match oldest_xact_start { + Some(oldest) => oldest.min(sample_s), + None => sample_s, + }; + lower + - chrono::Duration::seconds(CREATED_AT_FLOOR_SECS) + - chrono::Duration::seconds(FENCE_CLOCK_MARGIN_SECS) } -/// Run one full handshake and, on success, advance the fence. +/// Run one full handshake and record the resulting `(token, fence_wall)`. /// -/// Returns the new fence value for observability. `Ok(None)` means the -/// replica has not yet replayed past the sample; the fence is left as-is -/// (staleness will close it if this persists). -pub async fn probe_once( - writer: &PgPool, - replica: &PgPool, - fence: &ReplicaFence, -) -> Result>, ProbeError> { +/// On a same-epoch token regression (the writer was restored from a backup +/// that kept its epoch), the retained ring has already been cleared by +/// [`ReplicaFence::record`]; this additionally **rotates the epoch** on the +/// writer and records the rotated token, so a reader still serving the +/// pre-rewind timeline (whose old, higher token would otherwise satisfy +/// `token >= M`) fails the epoch check instead of proving stale coverage. +pub async fn probe_once(writer: &PgPool, fence: &ReplicaFence) -> Result { let sample = sample_writer(writer).await?; - if !replica_covers(replica, &sample.wal_lsn).await? { - return Ok(None); + let wall = fence_wall(sample.sampled_at, sample.oldest_xact_start); + match fence.record(sample.token, sample.epoch, sample.committed_at, wall) { + RecordOutcome::Recorded => Ok(TokenEntry { + token: sample.token, + committed_at: sample.committed_at, + fence_wall: wall, + }), + RecordOutcome::TokenRegression => { + tracing::warn!( + token = sample.token, + "replica heartbeat token regressed within its epoch (restore?); rotating epoch" + ); + // The rotation commit happens after this sample's activity scan, + // so the same three-bucket argument (and the same wall) holds + // for the rotated token. + let committed_at = Instant::now(); + let row = sqlx::query( + "UPDATE replica_heartbeat SET epoch = gen_random_uuid(), token = token + 1 \ + WHERE id = 1 RETURNING token, epoch", + ) + .fetch_optional(writer) + .await? + .ok_or(ProbeError::HeartbeatRowMissing)?; + let token: i64 = row.get("token"); + let epoch: Uuid = row.get("epoch"); + // A fresh epoch always clears and records; regression is + // impossible against an empty ring. + fence.record(token, epoch, committed_at, wall); + Ok(TokenEntry { + token, + committed_at, + fence_wall: wall, + }) + } } - let lower = match sample.oldest_xact_start { - Some(oldest) => oldest.min(sample.sampled_at), - None => sample.sampled_at, +} + +/// The Aurora **PostgreSQL** instance-identity function. Named once so the +/// capability probe and the observation query can never disagree — and +/// pinned by a unit test, because the MySQL-family spelling +/// (`aurora_server_id`) is a near-miss that would make the capability probe +/// cache a permanent `false` on real Aurora (42883) and silently strip the +/// instance id from canary evidence. +pub const AURORA_IDENTITY_FN: &str = "aurora_db_instance_identifier"; + +/// Whether this reader endpoint supports [`AURORA_IDENTITY_FN`] — probed +/// ONCE per process on a plain autocommit checkout, never inside a request +/// transaction (an undefined-function error would abort the transaction +/// and fail the proof). `Ok(false)` is the definitive "not Aurora" answer +/// (undefined_function, SQLSTATE 42883); transient errors surface as `Err` +/// so the caller can retry the probe on a later request instead of caching +/// a wrong answer. +pub async fn reader_supports_aurora_identity(conn: &mut PgConnection) -> Result { + match sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {AURORA_IDENTITY_FN}()" + ))) + .fetch_one(&mut *conn) + .await + { + Ok(_) => Ok(true), + Err(sqlx::Error::Database(e)) if e.code().as_deref() == Some("42883") => Ok(false), + Err(e) => Err(e), + } +} + +/// Observe the heartbeat on a specific reader session — the +/// connection-local half of the proof. Returns the observed token/epoch +/// plus the backend identity of the session for route-decision evidence: +/// `addr:port pid=N` (`local` on unix sockets), prefixed with the Aurora +/// instance id when `aurora` is set (only pass `true` after +/// [`reader_supports_aurora_identity`] confirmed it — the function +/// reference fails at parse time on plain Postgres). `None` when the row +/// is missing there (migration not yet replayed): fail closed. +pub async fn observe_heartbeat( + conn: &mut PgConnection, + aurora: bool, +) -> Result, sqlx::Error> { + const ADDR_PID: &str = "COALESCE(host(inet_server_addr()) || ':' || \ + inet_server_port()::text, 'local') || ' pid=' || pg_backend_pid()::text"; + let sql = if aurora { + format!( + "SELECT token, epoch, {AURORA_IDENTITY_FN}() || ' @ ' || {ADDR_PID} AS backend \ + FROM replica_heartbeat WHERE id = 1" + ) + } else { + format!( + "SELECT token, epoch, {ADDR_PID} AS backend \ + FROM replica_heartbeat WHERE id = 1" + ) }; - let new_fence = lower - - chrono::Duration::seconds(CREATED_AT_FLOOR_SECS) - - chrono::Duration::seconds(FENCE_CLOCK_MARGIN_SECS); - fence.advance(new_fence); - Ok(Some(new_fence)) + let row = sqlx::query(sqlx::AssertSqlSafe(sql)) + .fetch_optional(&mut *conn) + .await?; + Ok(row.map(|r| HeartbeatObservation { + token: r.get("token"), + epoch: r.get("epoch"), + backend: r.get("backend"), + })) } -/// Background probe loop: sample every `PROBE_INTERVAL`, close the fence on -/// any error. Runs for the life of the process. -pub async fn run_probe(writer: PgPool, replica: PgPool, fence: Arc) { +/// One reader-session heartbeat observation (see [`observe_heartbeat`]). +#[derive(Debug, Clone)] +pub struct HeartbeatObservation { + /// The token the session has replayed through. + pub token: i64, + /// The epoch the session observes — must match the ring's. + pub epoch: Uuid, + /// Backend identity of the observed session, so live evidence records + /// which reader served both proof and page. + pub backend: String, +} + +/// Background probe loop: commit a heartbeat token every `PROBE_INTERVAL`; +/// close the fence on any error. Runs for the life of the process. +pub async fn run_probe(writer: PgPool, fence: Arc) { let mut interval = tokio::time::interval(PROBE_INTERVAL); interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); loop { interval.tick().await; - match probe_once(&writer, &replica, &fence).await { - Ok(Some(_)) => {} - Ok(None) => { - // Replica behind the sample: leave the fence; staleness - // closes it if the replica stays behind. - tracing::debug!("replica fence: replay behind writer sample"); - } + match probe_once(&writer, &fence).await { + Ok(_) => {} Err(e) => { fence.close(); tracing::warn!(error = %e, "replica fence probe failed; fence closed"); @@ -515,14 +785,50 @@ mod tests { std::env::var("TEST_DATABASE_URL").unwrap_or_else(|_| TEST_DB_URL.into()) } + /// A private scratch database with migrations applied: the probe tests + /// mutate the singleton heartbeat row (rewind/rotate), which must never + /// race the shared dev database or each other. + async fn scratch_db() -> (PgPool, PgPool, String) { + let admin = PgPool::connect(&test_db_url()) + .await + .expect("connect admin"); + let name = format!("fence_probe_{}", uuid::Uuid::new_v4().simple()); + sqlx::query(sqlx::AssertSqlSafe(format!("CREATE DATABASE {name}"))) + .execute(&admin) + .await + .expect("create scratch db"); + let base = test_db_url(); + let idx = base.rfind('/').expect("db url has a path segment"); + let pool = PgPool::connect(&format!("{}/{}", &base[..idx], name)) + .await + .expect("connect scratch db"); + crate::migration::run_migrations(&pool) + .await + .expect("migrate scratch db"); + (admin, pool, name) + } + + async fn drop_scratch_db(admin: &PgPool, pool: PgPool, name: &str) { + pool.close().await; + let _ = sqlx::query(sqlx::AssertSqlSafe(format!( + "DROP DATABASE IF EXISTS {name} WITH (FORCE)" + ))) + .execute(admin) + .await; + } + #[test] - fn fence_starts_closed_and_opens_on_advance() { + fn fence_starts_closed_and_opens_on_record() { let fence = ReplicaFence::new(); assert!(fence.verified_through().is_none(), "must start closed"); assert!(!fence.covers(Utc::now() - chrono::Duration::days(365))); let ts = Utc::now(); - fence.advance(ts); + let epoch = Uuid::new_v4(); + assert_eq!( + fence.record(1, epoch, Instant::now(), ts), + RecordOutcome::Recorded + ); assert_eq!(fence.verified_through(), Some(ts)); assert!(fence.covers(ts - chrono::Duration::seconds(1))); assert!(fence.covers(ts), "boundary is inclusive"); @@ -537,19 +843,116 @@ mod tests { fn stale_fence_reads_as_closed() { let fence = ReplicaFence::new(); let ts = Utc::now(); - fence - .fence_micros - .store(ts.timestamp_micros(), Ordering::Relaxed); - // Last update older than the staleness budget. - let stale = (Utc::now() - - chrono::Duration::from_std(FENCE_STALENESS).expect("duration") - - chrono::Duration::seconds(1)) - .timestamp_micros(); - fence.updated_micros.store(stale, Ordering::Relaxed); + // Newest entry committed longer ago than the staleness budget. + let stale_instant = Instant::now() - (FENCE_STALENESS + Duration::from_secs(1)); + fence.record(1, Uuid::new_v4(), stale_instant, ts); assert!( fence.verified_through().is_none(), "a fence the probe stopped confirming must read as closed" ); + // heartbeat_age is deliberately ungated (observability). + assert!(fence.heartbeat_age().expect("entry retained") > FENCE_STALENESS); + } + + /// Resolve picks the greatest retained entry <= the observed token — + /// a lagged reader proves from an older wall, never from thin air. + #[test] + fn resolve_picks_greatest_retained_token_at_or_below_observation() { + let fence = ReplicaFence::new(); + let epoch = Uuid::new_v4(); + let base = Utc::now(); + for (token, secs) in [(10i64, 0i64), (20, 10), (30, 20)] { + fence.record( + token, + epoch, + Instant::now(), + base + chrono::Duration::seconds(secs), + ); + } + + // Exact hit. + assert_eq!(fence.resolve(20, epoch).proved().expect("proof").token, 20); + // Between entries: prove from the older one. + assert_eq!(fence.resolve(25, epoch).proved().expect("proof").token, 20); + // Ahead of everything retained: newest. + assert_eq!( + fence.resolve(1000, epoch).proved().expect("proof").token, + 30 + ); + // Behind everything retained: no proof. + assert_eq!( + fence.resolve(9, epoch), + ResolveOutcome::TokenBehind, + "token below ring fails closed" + ); + // Wrong epoch: no proof, regardless of token. + assert_eq!( + fence.resolve(1000, Uuid::new_v4()), + ResolveOutcome::EpochMismatch, + "epoch mismatch fails closed" + ); + } + + /// An epoch change clears the ring and starts a new one; a same-epoch + /// token regression clears the ring and reports the fault. + #[test] + fn record_epoch_change_resets_and_same_epoch_regression_fails() { + let fence = ReplicaFence::new(); + let epoch_a = Uuid::new_v4(); + let ts = Utc::now(); + fence.record(10, epoch_a, Instant::now(), ts); + fence.record(11, epoch_a, Instant::now(), ts); + + // New epoch, lower token: fine — new timeline, old proofs dropped. + let epoch_b = Uuid::new_v4(); + assert_eq!( + fence.record(3, epoch_b, Instant::now(), ts), + RecordOutcome::Recorded + ); + assert_eq!( + fence.resolve(11, epoch_a), + ResolveOutcome::EpochMismatch, + "entries from the old epoch must be gone" + ); + assert_eq!(fence.resolve(3, epoch_b).proved().expect("proof").token, 3); + + // Same epoch, non-increasing token: regression → cleared + reported. + assert_eq!( + fence.record(3, epoch_b, Instant::now(), ts), + RecordOutcome::TokenRegression + ); + assert!( + fence.verified_through().is_none(), + "ring cleared on regression" + ); + assert_eq!( + fence.resolve(i64::MAX, epoch_b), + ResolveOutcome::TokenBehind + ); + } + + /// The ring is bounded: old entries fall off and stop proving coverage. + #[test] + fn ring_capacity_evicts_oldest_entries() { + let fence = ReplicaFence::new(); + let epoch = Uuid::new_v4(); + let ts = Utc::now(); + for token in 0..(RING_CAPACITY as i64 + 10) { + fence.record(token, epoch, Instant::now(), ts); + } + assert_eq!( + fence.resolve(5, epoch), + ResolveOutcome::TokenBehind, + "evicted tokens must no longer prove coverage" + ); + assert_eq!( + fence + .resolve(i64::MAX, epoch) + .proved() + .expect("proof") + .token, + RING_CAPACITY as i64 + 9 + ); } /// The activity scan must (a) represent another session's open @@ -582,9 +985,14 @@ mod tests { oldest <= during.sampled_at, "xact_start precedes the sample that observed it" ); - // S is captured before the activity scan, L after: the sample's - // ordering invariant. + // S is captured before the activity scan, the token commit after: + // the sample's ordering invariant. assert!(during.sampled_at >= before.sampled_at); + assert!( + during.token > before.token, + "each sample must commit a strictly newer token" + ); + assert_eq!(during.epoch, before.epoch, "epoch is stable across samples"); tx.rollback().await.expect("rollback"); } @@ -641,21 +1049,140 @@ mod tests { .expect("drop role"); } - /// A primary (non-replica) database returns NULL from - /// `pg_last_wal_replay_lsn()`; the probe must fail closed, never - /// synthesize freshness. This is also the Aurora-observability guard: - /// if the reader endpoint hides replay LSNs, routing stays writer-only. + /// The Aurora PostgreSQL identity function name is exact — the + /// MySQL-family near-miss (`aurora_server_id`) would make the + /// capability probe cache a permanent false on real Aurora and + /// silently strip the instance id from canary evidence (Wren, delta + /// review of a472327). AWS reference: aurora_db_instance_identifier() + /// (Aurora PostgreSQL user guide; also awslabs/pg-collector). + #[test] + fn aurora_identity_function_name_is_the_postgres_one() { + assert_eq!(AURORA_IDENTITY_FN, "aurora_db_instance_identifier"); + } + + /// The Aurora identity capability probe must answer a definitive + /// `false` on plain Postgres (undefined_function), not error — and the + /// error path must not poison the connection for later statements. #[tokio::test] #[ignore = "requires Postgres"] - async fn probe_fails_closed_when_replica_lsn_unavailable() { + async fn aurora_identity_probe_reports_false_on_plain_postgres() { let pool = PgPool::connect(&test_db_url()).await.expect("connect"); + let mut conn = pool.acquire().await.expect("conn"); + assert!( + !reader_supports_aurora_identity(&mut conn) + .await + .expect("probe must not error on plain postgres"), + "plain postgres must report no aurora identity support" + ); + // The failed function lookup must not have wedged the session. + let one: i32 = sqlx::query_scalar("SELECT 1") + .fetch_one(&mut *conn) + .await + .expect("connection usable after probe"); + assert_eq!(one, 1); + } + + /// End-to-end probe against a real database: each probe commits a + /// strictly newer token, records a retained entry, and a session on the + /// same database observes a token/epoch that resolves that entry. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn probe_commits_tokens_and_sessions_prove_coverage() { + let (admin, pool, name) = scratch_db().await; + let fence = ReplicaFence::new(); + + let first = probe_once(&pool, &fence).await.expect("first probe"); + let second = probe_once(&pool, &fence).await.expect("second probe"); + assert!(second.token > first.token, "tokens strictly increase"); + assert!( + second.fence_wall >= first.fence_wall + || second.fence_wall + > first.fence_wall - chrono::Duration::seconds(FENCE_CLOCK_MARGIN_SECS), + "walls advance with the clock (modulo an open transaction)" + ); + + // A "reader" session on the same database observes at least the + // second token and proves the newest retained entry. + let mut conn = pool.acquire().await.expect("reader conn"); + let obs = observe_heartbeat(&mut conn, false) + .await + .expect("observe") + .expect("heartbeat row present"); + assert!(obs.token >= second.token); + assert!( + obs.backend.contains(" pid="), + "backend identity must carry the backend pid, got {:?}", + obs.backend + ); + // TCP fixtures also carry addr:port; unix-socket fixtures read 'local'. + assert!( + obs.backend.starts_with("local pid=") || obs.backend.contains(':'), + "backend identity must carry addr:port or 'local', got {:?}", + obs.backend + ); + let proof = fence + .resolve(obs.token, obs.epoch) + .proved() + .expect("proof resolves"); + assert_eq!(proof.token, second.token, "newest retained entry cited"); + + // An epoch nobody committed proves nothing. + assert_eq!( + fence.resolve(obs.token, Uuid::new_v4()), + ResolveOutcome::EpochMismatch + ); + + drop(conn); + drop_scratch_db(&admin, pool, &name).await; + } + + /// A same-epoch token rewind on the writer (restore adversary) must not + /// leave proofs standing: the probe rotates the epoch, so a reader still + /// on the pre-rewind timeline — observing a *higher* token under the old + /// epoch — fails the epoch check instead of proving stale coverage. + #[tokio::test] + #[ignore = "requires Postgres"] + async fn probe_rotates_epoch_on_same_epoch_token_regression() { + let (admin, pool, name) = scratch_db().await; let fence = ReplicaFence::new(); - fence.advance(Utc::now()); // pretend a previous handshake succeeded - // Using the primary as its own "replica": replay LSN is NULL. - let err = probe_once(&pool, &pool, &fence) + let before = probe_once(&pool, &fence).await.expect("probe"); + let mut conn = pool.acquire().await.expect("conn"); + let old_epoch = observe_heartbeat(&mut conn, false) + .await + .expect("observe") + .expect("row") + .epoch; + + // Rewind the token in place, keeping the epoch: the restore shape. + sqlx::query("UPDATE replica_heartbeat SET token = 0 WHERE id = 1") + .execute(&pool) + .await + .expect("rewind token"); + + let after = probe_once(&pool, &fence).await.expect("recovery probe"); + // The pre-rewind observation must no longer prove anything. + assert_eq!( + fence.resolve(before.token, old_epoch), + ResolveOutcome::EpochMismatch, + "old-epoch observations must fail closed after rotation" + ); + // A fresh observation on the new timeline proves the rotated entry. + let obs = observe_heartbeat(&mut conn, false) .await - .expect_err("NULL replay LSN must be an error"); - assert!(matches!(err, ProbeError::ReplicaLsnUnavailable)); + .expect("observe") + .expect("row"); + assert_ne!(obs.epoch, old_epoch, "epoch rotated"); + assert_eq!( + fence + .resolve(obs.token, obs.epoch) + .proved() + .expect("proof") + .token, + after.token + ); + + drop(conn); + drop_scratch_db(&admin, pool, &name).await; } } diff --git a/crates/buzz-db/src/thread.rs b/crates/buzz-db/src/thread.rs index 3f92212dd5..007677e258 100644 --- a/crates/buzz-db/src/thread.rs +++ b/crates/buzz-db/src/thread.rs @@ -349,6 +349,30 @@ pub async fn get_thread_replies( depth_limit: Option, limit: u32, cursor: Option<&[u8]>, +) -> Result> { + let mut conn = pool.acquire().await?; + get_thread_replies_on( + &mut conn, + community_id, + root_event_id, + depth_limit, + limit, + cursor, + ) + .await +} + +/// [`get_thread_replies`] on a specific session — the replica-routing path +/// runs the page on the exact reader connection whose heartbeat observation +/// proved coverage (the proof is connection-local; a different pooled +/// session may sit at a different replay position). +pub(crate) async fn get_thread_replies_on( + conn: &mut sqlx::PgConnection, + community_id: CommunityId, + root_event_id: &[u8], + depth_limit: Option, + limit: u32, + cursor: Option<&[u8]>, ) -> Result> { // Decode cursor bytes -> keyset (timestamp, optional event_id) for the // WHERE condition. Layout: 8-byte BE i64 seconds, then the raw event_id. @@ -445,7 +469,7 @@ pub async fn get_thread_replies( } q = q.bind(limit as i32); - let rows = q.fetch_all(pool).await?; + let rows = q.fetch_all(&mut *conn).await?; let mut replies = Vec::with_capacity(rows.len()); for row in rows { @@ -569,6 +593,31 @@ pub async fn get_channel_window( limit: u32, cursor: Option<(DateTime, Vec)>, kind_filter: Option<&[u32]>, +) -> Result { + let mut conn = pool.acquire().await?; + get_channel_window_on( + &mut conn, + community_id, + channel_id, + limit, + cursor, + kind_filter, + ) + .await +} + +/// [`get_channel_window`] on a specific session — the replica-routing path +/// runs the page (and its participants batch) on the exact reader connection +/// whose heartbeat observation proved coverage (the proof is +/// connection-local; a different pooled session may sit at a different +/// replay position). +pub(crate) async fn get_channel_window_on( + conn: &mut sqlx::PgConnection, + community_id: CommunityId, + channel_id: Uuid, + limit: u32, + cursor: Option<(DateTime, Vec)>, + kind_filter: Option<&[u32]>, ) -> Result { let mut param_idx = 3u32; // $1 is community_id, $2 is channel_id let mut sql = String::from( @@ -637,7 +686,7 @@ pub async fn get_channel_window( // The +1 probe row is the server-internal has_more evidence. q = q.bind(limit as i64 + 1); - let mut db_rows = q.fetch_all(pool).await?; + let mut db_rows = q.fetch_all(&mut *conn).await?; let has_more = db_rows.len() > limit as usize; db_rows.truncate(limit as usize); @@ -722,7 +771,7 @@ pub async fn get_channel_window( ) .bind(community_id.as_uuid()) .bind(&roots) - .fetch_all(pool) + .fetch_all(&mut *conn) .await?; let mut by_root: std::collections::HashMap, Vec>> = diff --git a/crates/buzz-relay/src/api/bridge.rs b/crates/buzz-relay/src/api/bridge.rs index 2e00d6bd2f..10461d8d46 100644 --- a/crates/buzz-relay/src/api/bridge.rs +++ b/crates/buzz-relay/src/api/bridge.rs @@ -462,9 +462,9 @@ async fn handle_channel_window_filter( .as_ref() .map(|ks| ks.iter().map(|k| k.as_u16() as u32).collect()); - let window = state + let (window, mut session) = state .db - .get_channel_window( + .get_channel_window_with_session( tenant.community(), ch_id, limit, @@ -485,7 +485,12 @@ async fn handle_channel_window_filter( // 2. Aux closure: reactions/deletions/edits targeting retained rows, plus // deletions targeting those aux events (the transitive second hop). - // One round trip for the client instead of an #e fan-out. + // One round trip for the client instead of an #e fan-out. Runs in the + // SAME request transaction that served the window: when the page came + // from a proved replica session, the heartbeat observation anchored a + // REPEATABLE READ snapshot, so the aux hops see exactly the state the + // proof covered — another pooled session (or even another autocommit + // statement) could sit at a different replay position. if extension_flag(raw, "include_aux") && !row_ids_hex.is_empty() { let mut seen_aux: std::collections::HashSet = std::collections::HashSet::new(); @@ -495,8 +500,7 @@ async fn handle_channel_window_filter( aux_query.kinds = Some(hop_kinds.iter().map(|k| *k as i32).collect()); aux_query.e_tags = Some(std::mem::take(&mut hop_ids)); aux_query.limit = Some(1000); - let aux_events = state - .db + let aux_events = session .query_events(&aux_query) .await .map_err(|e| internal_error(&format!("window aux error: {e}")))?; @@ -1083,7 +1087,8 @@ async fn query_events_authed( let type_events = match canonical { "mentions" => state .db - .query_feed_mentions( + .query_feed_mentions_routed( + "bridge_feed", tenant.community(), &pubkey_bytes, &accessible_channels, @@ -1094,7 +1099,8 @@ async fn query_events_authed( .map_err(|e| internal_error(&format!("feed mentions error: {e}")))?, "needs_action" => state .db - .query_feed_needs_action( + .query_feed_needs_action_routed( + "bridge_feed", tenant.community(), &pubkey_bytes, &accessible_channels, @@ -1105,7 +1111,13 @@ async fn query_events_authed( .map_err(|e| internal_error(&format!("feed needs_action error: {e}")))?, "activity" => state .db - .query_feed_activity(tenant.community(), &accessible_channels, since, remaining) + .query_feed_activity_routed( + "bridge_feed", + tenant.community(), + &accessible_channels, + since, + remaining, + ) .await .map_err(|e| internal_error(&format!("feed activity error: {e}")))?, _ => continue, @@ -1271,7 +1283,7 @@ async fn query_events_authed( let db = state.db.clone(); let mut catchall_results = stream::iter(catchall_queries.into_iter().map(|(idx, query)| { let db = db.clone(); - async move { (idx, db.query_events(&query).await) } + async move { (idx, db.query_events_routed("bridge_query", &query).await) } })) .buffered(crate::handlers::req::FILTER_QUERY_CONCURRENCY); @@ -1476,7 +1488,7 @@ async fn count_events_authed( && !needs_result_gated_filtering && !needs_persona_filtering { - match state.db.count_events(&query).await { + match state.db.count_events_routed("bridge_count", &query).await { Ok(n) => total += n as u64, Err(e) => { return Err(internal_error(&format!("count error: {e}"))); @@ -1486,7 +1498,11 @@ async fn count_events_authed( // Fallback: query + post-filter for non-pushable constraints. let mut q = query; crate::handlers::req::apply_count_fallback_limit(&mut q); - match state.db.query_events(&q).await { + match state + .db + .query_events_routed_bounded("bridge_count_fallback", &q) + .await + { Ok(stored_events) => { if crate::handlers::req::count_fallback_exceeded(stored_events.len()) { metrics::counter!("buzz_count_fallback_rejections_total").increment(1); @@ -1543,7 +1559,7 @@ async fn count_events_authed( && !needs_persona_filtering { query.limit = None; - match state.db.count_events(&query).await { + match state.db.count_events_routed("bridge_count", &query).await { Ok(n) => total += n as u64, Err(e) => { return Err(internal_error(&format!("count error: {e}"))); @@ -1552,7 +1568,11 @@ async fn count_events_authed( } else { // Fallback: query a bounded candidate set + post-filter. crate::handlers::req::apply_count_fallback_limit(&mut query); - match state.db.query_events(&query).await { + match state + .db + .query_events_routed_bounded("bridge_count_fallback", &query) + .await + { Ok(stored_events) => { if crate::handlers::req::count_fallback_exceeded(stored_events.len()) { metrics::counter!("buzz_count_fallback_rejections_total").increment(1); @@ -1725,7 +1745,7 @@ async fn handle_bridge_search( let id_refs: Vec<&[u8]> = hit_ids.iter().map(|b| b.as_slice()).collect(); let stored_events = state .db - .get_events_by_ids(tenant.community(), &id_refs) + .get_events_by_ids_routed("bridge_search_hydrate", tenant.community(), &id_refs) .await .map_err(|e| internal_error(&format!("search fetch error: {e}")))?; diff --git a/crates/buzz-relay/src/config.rs b/crates/buzz-relay/src/config.rs index e494355736..85a0ca2efe 100644 --- a/crates/buzz-relay/src/config.rs +++ b/crates/buzz-relay/src/config.rs @@ -56,6 +56,10 @@ pub struct Config { /// Optional read-replica connection URL (e.g. an Aurora `cluster-ro-` /// endpoint). Unset means all reads stay on the writer. pub read_database_url: Option, + /// Replica read budget `B` in milliseconds (`BUZZ_REPLICA_READ_MAX_AGE_MS`). + /// `0` (the default) disables bounded-staleness replica routing; see + /// [`buzz_db::DbConfig::replica_read_max_age_ms`]. + pub replica_read_max_age_ms: u64, /// Redis connection URL used by the pub/sub manager. pub redis_url: String, /// Maximum connections in the shared Redis pool. Defaults to 16. @@ -72,6 +76,11 @@ pub struct Config { /// the per-pod pool and requests fail on acquire timeout while the /// database sits idle. pub db_pool_size: u32, + /// Maximum connections in the Postgres read-replica pool + /// (`BUZZ_DB_READ_POOL_SIZE`). Defaults to `db_pool_size`. Sized + /// independently so reader capacity can be tuned against the replica's + /// headroom without touching the writer pool. + pub db_read_pool_size: Option, /// Public WebSocket URL of this relay, advertised in NIP-11. pub relay_url: String, /// Public WebSocket URL of the dedicated device-pairing relay, when configured. @@ -423,6 +432,27 @@ impl Config { .map(|v| v.trim().to_string()) .filter(|v| !v.is_empty()); + // The old seconds-denominated name is a hard startup error, not an + // alias: silently honouring it would mean 1000x the intended budget. + if std::env::var("BUZZ_REPLICA_HEAD_MAX_AGE_SECS").is_ok() { + return Err(ConfigError::InvalidValue( + "BUZZ_REPLICA_HEAD_MAX_AGE_SECS was renamed to BUZZ_REPLICA_READ_MAX_AGE_MS \ + (note: milliseconds, not seconds); refusing to start" + .to_string(), + )); + } + + // Replica read budget: 0 = off (the rollout default), so this is a + // non-negative parse, unlike `positive_u64_from_env`. + let replica_read_max_age_ms = match std::env::var("BUZZ_REPLICA_READ_MAX_AGE_MS") { + Ok(raw) => raw.trim().parse::().map_err(|_| { + ConfigError::InvalidValue( + "BUZZ_REPLICA_READ_MAX_AGE_MS must be a non-negative integer".to_string(), + ) + })?, + Err(_) => 0, + }; + let redis_url = std::env::var("REDIS_URL").unwrap_or_else(|_| "redis://localhost:6379".to_string()); @@ -438,6 +468,11 @@ impl Config { .filter(|&v| v > 0) .unwrap_or(50); + let db_read_pool_size = std::env::var("BUZZ_DB_READ_POOL_SIZE") + .ok() + .and_then(|v| v.parse::().ok()) + .filter(|&v| v > 0); + let relay_url = std::env::var("RELAY_URL").unwrap_or_else(|_| "ws://localhost:3000".to_string()); @@ -898,9 +933,11 @@ impl Config { bind_addr, database_url, read_database_url, + replica_read_max_age_ms, redis_url, redis_pool_size, db_pool_size, + db_read_pool_size, relay_url, pairing_relay_url, max_connections, @@ -1121,6 +1158,35 @@ mod tests { assert_eq!(junk, 50, "unparsable value must fall back to the default"); } + #[test] + fn db_read_pool_size_env_override_and_invalid_fallback() { + let _guard = ENV_MUTEX.lock().unwrap(); + let previous = std::env::var_os("BUZZ_DB_READ_POOL_SIZE"); + + std::env::remove_var("BUZZ_DB_READ_POOL_SIZE"); + let unset = Config::from_env().expect("config").db_read_pool_size; + + std::env::set_var("BUZZ_DB_READ_POOL_SIZE", "40"); + let overridden = Config::from_env().expect("config").db_read_pool_size; + + std::env::set_var("BUZZ_DB_READ_POOL_SIZE", "0"); + let zero = Config::from_env().expect("config").db_read_pool_size; + + std::env::set_var("BUZZ_DB_READ_POOL_SIZE", "not-a-number"); + let junk = Config::from_env().expect("config").db_read_pool_size; + + if let Some(value) = previous { + std::env::set_var("BUZZ_DB_READ_POOL_SIZE", value); + } else { + std::env::remove_var("BUZZ_DB_READ_POOL_SIZE"); + } + + assert_eq!(unset, None, "unset must inherit the writer pool sizing"); + assert_eq!(overridden, Some(40)); + assert_eq!(zero, None, "zero must fall back to inheriting"); + assert_eq!(junk, None, "unparsable value must fall back to inheriting"); + } + #[test] fn read_database_url_unset_or_blank_is_none() { let _guard = ENV_MUTEX.lock().unwrap(); @@ -1149,6 +1215,58 @@ mod tests { ); } + #[test] + fn replica_read_max_age_defaults_off_and_rejects_junk() { + let _guard = ENV_MUTEX.lock().unwrap(); + let previous = std::env::var_os("BUZZ_REPLICA_READ_MAX_AGE_MS"); + let previous_old = std::env::var_os("BUZZ_REPLICA_HEAD_MAX_AGE_SECS"); + std::env::remove_var("BUZZ_REPLICA_HEAD_MAX_AGE_SECS"); + + std::env::remove_var("BUZZ_REPLICA_READ_MAX_AGE_MS"); + let unset = Config::from_env().expect("config").replica_read_max_age_ms; + + std::env::set_var("BUZZ_REPLICA_READ_MAX_AGE_MS", "1000"); + let set = Config::from_env().expect("config").replica_read_max_age_ms; + + std::env::set_var("BUZZ_REPLICA_READ_MAX_AGE_MS", "0"); + let zero = Config::from_env().expect("config").replica_read_max_age_ms; + + std::env::set_var("BUZZ_REPLICA_READ_MAX_AGE_MS", "soon"); + let junk = Config::from_env(); + + // The retired seconds-denominated name must be a hard startup + // error even alongside a valid new-name value: silently ignoring + // it (or honouring it) would mean 1000x the intended budget. + std::env::set_var("BUZZ_REPLICA_READ_MAX_AGE_MS", "1000"); + std::env::set_var("BUZZ_REPLICA_HEAD_MAX_AGE_SECS", "5"); + let old_name = Config::from_env(); + + std::env::remove_var("BUZZ_REPLICA_HEAD_MAX_AGE_SECS"); + if let Some(value) = previous { + std::env::set_var("BUZZ_REPLICA_READ_MAX_AGE_MS", value); + } else { + std::env::remove_var("BUZZ_REPLICA_READ_MAX_AGE_MS"); + } + if let Some(value) = previous_old { + std::env::set_var("BUZZ_REPLICA_HEAD_MAX_AGE_SECS", value); + } + + assert_eq!(unset, 0, "replica read routing must default off"); + assert_eq!(set, 1000); + assert_eq!(zero, 0, "explicit 0 is off"); + assert!( + junk.is_err(), + "an unparsable budget must fail loudly, not silently disable" + ); + match old_name { + Err(ConfigError::InvalidValue(message)) => assert!( + message.contains("BUZZ_REPLICA_READ_MAX_AGE_MS"), + "the error must name the replacement env var, got: {message}" + ), + other => panic!("old env name must hard-fail startup, got {other:?}"), + } + } + #[test] fn audit_logging_defaults_on_and_accepts_explicit_off() { let _guard = ENV_MUTEX.lock().unwrap(); diff --git a/crates/buzz-relay/src/handlers/count.rs b/crates/buzz-relay/src/handlers/count.rs index 614e54d7a0..dfb44e152f 100644 --- a/crates/buzz-relay/src/handlers/count.rs +++ b/crates/buzz-relay/src/handlers/count.rs @@ -173,7 +173,7 @@ pub async fn handle_count( && !needs_result_gated_filtering && !needs_persona_filtering { - match state.db.count_events(&query).await { + match state.db.count_events_routed("count_req", &query).await { Ok(n) => total += n as u64, Err(e) => { conn.send(RelayMessage::closed(&sub_id, &format!("error: {e}"))); @@ -184,7 +184,11 @@ pub async fn handle_count( // Fallback: query + post-filter for non-pushable constraints. let mut q = query; super::req::apply_count_fallback_limit(&mut q); - match state.db.query_events(&q).await { + match state + .db + .query_events_routed_bounded("count_req_fallback", &q) + .await + { Ok(stored_events) => { if super::req::count_fallback_exceeded(stored_events.len()) { metrics::counter!("buzz_count_fallback_rejections_total").increment(1); @@ -243,7 +247,7 @@ pub async fn handle_count( && !needs_persona_filtering { query.limit = None; // COUNT doesn't need a row limit - match state.db.count_events(&query).await { + match state.db.count_events_routed("count_req", &query).await { Ok(n) => total += n as u64, Err(e) => { conn.send(RelayMessage::closed(&sub_id, &format!("error: {e}"))); @@ -253,7 +257,11 @@ pub async fn handle_count( } else { // Fallback: query a bounded candidate set + post-filter. super::req::apply_count_fallback_limit(&mut query); - match state.db.query_events(&query).await { + match state + .db + .query_events_routed_bounded("count_req_fallback", &query) + .await + { Ok(stored_events) => { if super::req::count_fallback_exceeded(stored_events.len()) { metrics::counter!("buzz_count_fallback_rejections_total").increment(1); diff --git a/crates/buzz-relay/src/handlers/req.rs b/crates/buzz-relay/src/handlers/req.rs index d3ddd3e5d3..51400452d7 100644 --- a/crates/buzz-relay/src/handlers/req.rs +++ b/crates/buzz-relay/src/handlers/req.rs @@ -310,7 +310,7 @@ pub async fn handle_req( |(idx, per_filter_channel, params)| { let db = db.clone(); async move { - let filter_events = db.query_events(¶ms).await; + let filter_events = db.query_events_routed("req_historical", ¶ms).await; (idx, per_filter_channel, filter_events) } }, @@ -629,7 +629,7 @@ async fn handle_search_req( let id_refs: Vec<&[u8]> = hit_ids.iter().map(|b| b.as_slice()).collect(); let events = match state .db - .get_events_by_ids(tenant.community(), &id_refs) + .get_events_by_ids_routed("req_search_hydrate", tenant.community(), &id_refs) .await { Ok(evs) => evs, diff --git a/crates/buzz-relay/src/main.rs b/crates/buzz-relay/src/main.rs index 9e6ca828e0..799cf9cf60 100644 --- a/crates/buzz-relay/src/main.rs +++ b/crates/buzz-relay/src/main.rs @@ -166,7 +166,9 @@ async fn main() -> anyhow::Result<()> { let db_config = DbConfig { database_url: config.database_url.clone(), read_database_url: config.read_database_url.clone(), + replica_read_max_age_ms: config.replica_read_max_age_ms, max_connections: config.db_pool_size, + read_max_connections: config.db_read_pool_size, ..DbConfig::default() }; let db = Db::new(&db_config).await.map_err(|e| { @@ -174,7 +176,11 @@ async fn main() -> anyhow::Result<()> { anyhow::anyhow!("DB connection failed: {e}") })?; if db.has_read_pool() { - info!("Postgres connected (writer + read replica)"); + info!("Postgres connected (writer + lazy read replica pool)"); + // Reader-down at boot must not crash or block the relay; this warn-only + // ping is the sole boot-time visibility that the replica is unreachable + // (the lazy pool with min_connections=0 dials nothing until first use). + db.spawn_read_pool_boot_ping(); } else { info!("Postgres connected"); } @@ -999,6 +1005,12 @@ async fn main() -> anyhow::Result<()> { metrics::gauge!("buzz_db_replica_fence_open").set(0.0); } } + // Probe liveness, ungated by staleness: how long since + // the probe last committed a heartbeat token. + if let Some(age) = pool_state.db.fence().heartbeat_age() { + metrics::gauge!("buzz_db_replica_heartbeat_age_seconds") + .set(age.as_secs_f64()); + } } let rs = pool_state.redis_pool.status(); diff --git a/migrations/0026_replica_heartbeat.sql b/migrations/0026_replica_heartbeat.sql new file mode 100644 index 0000000000..278be4d87a --- /dev/null +++ b/migrations/0026_replica_heartbeat.sql @@ -0,0 +1,38 @@ +-- Replica heartbeat: a portable read-side freshness observation for the +-- replica fence (see crates/buzz-db/src/replica_fence.rs). +-- +-- Why: the fence's ordered writer-side proof previously ended in a WAL-LSN +-- comparison (`pg_last_wal_replay_lsn() >= L`), which Aurora's reader +-- endpoints do not expose — the fence therefore never opened on Aurora, by +-- design (fail closed). This table replaces only that read-side observation: +-- the probe commits a monotonically increasing `token` AFTER the ordered +-- writer scan (clock sample -> oldest-xact guard), and a reader session that +-- observes token >= M has, by WAL/storage replay order, also replayed every +-- commit that preceded M. The commit-time floor guard (migration 0021) and +-- the writer-side scan remain load-bearing and unchanged. +-- +-- Shape: exactly one row, enforced by the CHECK'd primary key. Every relay +-- pod's probe increments the same row; the single-row UPDATE is the +-- serialization point that makes tokens globally commit-ordered, which is +-- what lets a pod prove coverage from the greatest token it retained that is +-- <= the token a reader session observes (multi-pod safety). +-- +-- `epoch` detects resets: a restore/re-seed that rolls `token` backwards +-- must never let a stale retained token masquerade as fresh coverage. +-- Readers validate the observed epoch against the epoch retained with each +-- token; a mismatch fails closed (route to writer). +-- +-- Not an events row: exempt from the created_at floor guard by construction, +-- and deliberately deployment-global (no community_id) — it describes the +-- replication topology, not tenant data. + +CREATE TABLE replica_heartbeat ( + id smallint PRIMARY KEY CHECK (id = 1), + epoch uuid NOT NULL DEFAULT gen_random_uuid(), + token bigint NOT NULL DEFAULT 0 +); + +INSERT INTO replica_heartbeat (id) VALUES (1); + +INSERT INTO _operator_global_tables (table_name, reason) VALUES + ('replica_heartbeat', 'single-row replication freshness token; describes deployment topology, never tenant data'); diff --git a/schema/schema.sql b/schema/schema.sql index f5f32cc3e3..5980695d32 100644 --- a/schema/schema.sql +++ b/schema/schema.sql @@ -1051,3 +1051,23 @@ INSERT INTO _operator_global_tables (table_name, reason) VALUES ('push_gateway_endpoint_quotas', 'public gateway endpoint abuse ceilings span relay communities'), ('push_gateway_delivery_auth_replays', 'public gateway signed-event replay admission spans relay communities'), ('push_gateway_delivery_request_replays', 'public gateway stable request-id admission spans relay communities'); + +-- ── Replica heartbeat (read-replica freshness fence) ───────────────────────── +-- Portable read-side freshness observation for the replica fence (see +-- crates/buzz-db/src/replica_fence.rs and migrations/0026). Exactly one row; +-- the single-row token UPDATE is the serialization point that makes tokens +-- globally commit-ordered across relay pods. `epoch` detects token resets +-- (restore/re-seed) so a stale retained token can never masquerade as fresh +-- coverage. Deployment-global by design: describes replication topology, +-- never tenant data. + +CREATE TABLE replica_heartbeat ( + id smallint PRIMARY KEY CHECK (id = 1), + epoch uuid NOT NULL DEFAULT gen_random_uuid(), + token bigint NOT NULL DEFAULT 0 +); + +INSERT INTO replica_heartbeat (id) VALUES (1); + +INSERT INTO _operator_global_tables (table_name, reason) VALUES + ('replica_heartbeat', 'single-row replication freshness token; describes deployment topology, never tenant data'); From eda5dc91d137cc3b1b0e512a7cd582b990f11f67 Mon Sep 17 00:00:00 2001 From: Brian Charbonneau Date: Thu, 30 Jul 2026 00:13:50 -0700 Subject: [PATCH 31/32] docs(desktop): add private CA release lifecycle Signed-off-by: Brian Charbonneau --- .../private-ca-desktop-release-lifecycle.md | 206 ++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 docs/operations/private-ca-desktop-release-lifecycle.md diff --git a/docs/operations/private-ca-desktop-release-lifecycle.md b/docs/operations/private-ca-desktop-release-lifecycle.md new file mode 100644 index 0000000000..f15681c258 --- /dev/null +++ b/docs/operations/private-ca-desktop-release-lifecycle.md @@ -0,0 +1,206 @@ +# Buzz private-CA desktop release lifecycle + +## Purpose + +This fork-only operational workflow publishes an Apple Silicon Buzz Desktop +build that trusts certificate authorities installed in the macOS platform trust +store. It exists while the upstream solution is under coordination in +[BrianInAz/buzz#1](https://github.com/BrianInAz/buzz/issues/1). + +The implementation is intentionally not an upstream release lane. It must not +publish a general `latest` release, inject upstream signing material, or change +the upstream auto-update path. + +## Security boundaries + +- The human private identity remains on the client. It must never enter Git, + Actions, Vault, AWX, a server, logs, screenshots, or release assets. +- The public fork contains no private CA material, private IP addresses, or + environment-specific credentials. +- The native connector continues normal host name, certificate-chain, and + expiry validation. It never uses insecure TLS or relay-specific exceptions. +- Persistent homelab runners are never attached to this public repository. +- A GitHub Actions issue event is untrusted input. Only the repository owner + can approve an immutable, verified upstream release build. + +## Preserved validated patch + +The current implementation is preserved independently of this automation: + +- Branch: `fix/macos-private-ca-websocket` +- Commit: `6d03a38da5e3402bf97df1b3c46152887eb3778e` +- Subject: `fix(desktop): trust platform CAs for native websockets` + +It uses `rustls-platform-verifier` with an explicit Rustls connector and routes +the primary relay, pairing, and huddle WebSocket paths through it. The commit +has the required DCO sign-off. Do not rewrite it. A future upstream PR starts +from then-current `upstream/main`, carries a DCO sign-off on every commit, and +is opened only when the upstream coordination issue asks for it or competing +work is withdrawn. + +## Repository layout and Git workflow + +Permanent checkouts are recreated from the fork, never copied from a temporary +worktree: + +```text +/Users/b/Code/buzz/human +/Users/b/Code/buzz/codex +``` + +`origin` is `BrianInAz/buzz`; `upstream` is `block/buzz`. + +The fork follows upstream's main-oriented contribution model for fork-only +automation. Work starts from verified `upstream/main` on +`codex/private-ca-release-automation`, is pushed normally, and is reviewed in +a PR to `BrianInAz/buzz:main`. This keeps future upstream contribution branches +clean. Never force-push. + +The private WSS workflow belongs in `BjzyLabs/homelab-playbooks` and follows +that repository's normal `develop` then targeted `main` promotion flow. + +## Durable task state + +The private operational task is Beads `tailscale-vault-of5-213`, a child of +`tailscale-vault-of5-211.13`. Every stopping point records branch, SHA, tests, +URLs, blockers, and the exact next action there. + +Before resuming, another agent must read, in order: + +1. This document. +2. Beads `tailscale-vault-of5-213` and its parent. +3. [BrianInAz/buzz#1](https://github.com/BrianInAz/buzz/issues/1). +4. The current branch, status, remotes, upstream activity, and CI results. + +If this document is absent, do not reconstruct the process from temporary +directories or stale chat transcripts; request the approved plan. + +## Daily release monitor + +The fork workflow runs daily at 15:05 UTC and by manual dispatch. It observes +new stable desktop releases in `block/buzz`; prereleases require explicit +manual selection. + +For a new immutable upstream release, it records the tag, source SHA, +publication time, release notes, comparison link, private-patch applicability, +and any explicitly recorded upstream trust fix. It creates exactly one assigned +issue titled `[Buzz update] vX.Y.Z available` with labels: + +```text +buzz-update +build-approved +skip +remediation-required +built +accepted +``` + +The ticket is the approval plane, not a generic notification. It provides a +concise changelog and a BUILD/SKIP decision. Repeated scheduled runs must not +open duplicates. + +`build-approved` is accepted only when the actor is exactly `BrianInAz`, the +issue was generated by the monitor, the source tag still resolves to the +recorded SHA, and the pinned patch applies cleanly. Public comments and issue +text never become command input. `skip` closes the ticket without a build. A +conflict or failed preflight applies `remediation-required` and publishes no +package. + +## Hosted validation and package + +Standard GitHub-hosted runners are used for bounded, reproducible work. They +are currently unmetered for this public repository, but concurrency, queueing, +cache, artifact, and fair-use limits still apply. Larger billable runners are +not used. + +The approved build has three gates: + +1. Ubuntu checks out the exact upstream tag, applies the pinned patch without + committing it, then runs formatting, Clippy, focused connector tests, and + `just ci`. +2. macOS arm64 runs a hermetic platform-trust test. It creates a disposable CA + and localhost WSS server, proves the connection fails before trust is added, + succeeds after adding that CA to a temporary macOS keychain, then removes + all temporary trust material. It covers primary, pairing, and huddle paths. +3. macOS arm64 packages the exact source using existing upstream sidecar and + Tauri conventions. The package has no updater keys, no notarization, no + Apple Developer signing identity, and no private infrastructure values. + The updater remains disabled. The app is ad-hoc signed and verified before + the DMG is rebuilt. + +Publish a fork prerelease named `buzz-private-ca-vX.Y.Z-rN`, never the general +latest release. Include the arm64 DMG, SHA256SUMS, applied patch, test summary, +and a manifest containing upstream tag/SHA, patch SHA, result tree SHA, tool +versions, runner architecture, and workflow URL. Add a GitHub build-provenance +attestation. Durable packages belong in the prerelease, not a short-lived +Actions artifact. + +## Private WSS gate + +The public workflow never receives access to the private endpoint. An approved +operator triggers a private `homelab-playbooks` workflow from the authenticated +Mac after the prerelease is available. + +That workflow runs only on existing `ghRunner` selected through a dedicated +`buzz-private-wss` label. Do not add Tailscale: the runner already has the +required private route and CA trust. + +It validates the exact source revision against `wss://buzz.bjzy.me` without an +identity key: + +- trusted HTTPS readiness returns 200; +- the WSS upgrade succeeds over TCP 443; +- the opt-in Rust connector test succeeds with `BUZZ_TEST_WSS_URL`; +- no backend or monitoring port is used. + +A private-gate failure blocks local installation. It is not repaired by ACL +expansion, alternate credentials, or TLS bypasses. + +## State-preserving Mac installation + +The current state may include two open Buzz windows: + +- the official release instance using `xyz.block.buzz.app` and production + application state; +- the working isolated development instance using a `.dev` identifier. + +Before installation, inventory process paths and bundle identifiers without +reading identity material. Keep both windows open during hosted CI. Once the +private gate is green, download the DMG and SHA256SUMS to `/Users/b/Downloads`, +verify checksum and provenance, and archive the current official bundle under +`/Users/b/Applications/Buzz Rollback//Buzz.app`. + +Do not modify Application Support, WebKit state, Keychain entries, identity +keys, community state, or history. Gracefully quit only the official release +instance, retain the working dev instance as rollback control, and replace the +canonical release bundle at its verified existing path. + +The build is ad-hoc signed rather than notarized. Open it using Finder's Open +action or macOS Privacy & Security's Open Anyway. Never disable Gatekeeper +globally. An Apple Developer account is not required for this personal build, +but each new build can require a local approval. + +Acceptance requires the canonical release app to reuse the existing identity +and community without another key import, then prove connection, owner status, +history load, message send, full quit, relaunch, automatic reconnect, and +history restoration. Only after this passes may the isolated development app be +closed normally. + +## Rollback and retirement + +- Failed source or hosted validation: publish nothing and mark the ticket + `remediation-required`. +- Failed private gate: do not install. +- Failed local acceptance: quit the patched app, restore the archived official + bundle, and prove it launches; keep the dev instance available. +- When a release contains a maintainer-recognized upstream fix, test the + official signed DMG through the same lifecycle before retiring this process. + Close issue #1 only after the chosen upstream outcome is recorded. + +## Documentation and completion + +Document the public process here, the private runner process in +`homelab-playbooks`, and the redacted operational procedure in Notion. Do not +claim completion until the monitor, BUILD/SKIP paths, hosted validation, +prerelease, private WSS gate, state-preserving install, restart/history +acceptance, rollback bundle, Beads evidence, and documentation are all green. From 01c378da9d650610fabcb6d7e6f05a08488639bb Mon Sep 17 00:00:00 2001 From: Brian Charbonneau Date: Thu, 30 Jul 2026 00:24:36 -0700 Subject: [PATCH 32/32] feat(release): automate private CA desktop builds Signed-off-by: Brian Charbonneau --- .github/workflows/ci.yml | 2 + .github/workflows/private-ca-release.yml | 225 ++++++++++++++++++++ scripts/test-private-ca-release-contract.sh | 62 ++++++ 3 files changed, 289 insertions(+) create mode 100644 .github/workflows/private-ca-release.yml create mode 100755 scripts/test-private-ca-release-contract.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d4826d985f..7edf5a7913 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,8 @@ jobs: - '.github/workflows/ci.yml' - name: Release workflow source contract run: scripts/test-release-ref-contract.sh + - name: Private CA release workflow contract + run: scripts/test-private-ca-release-contract.sh - name: Mobile release contract run: | scripts/test-mobile-release-contract.sh diff --git a/.github/workflows/private-ca-release.yml b/.github/workflows/private-ca-release.yml new file mode 100644 index 0000000000..c25e0ca7d9 --- /dev/null +++ b/.github/workflows/private-ca-release.yml @@ -0,0 +1,225 @@ +--- +# yamllint disable rule:line-length +# yamllint disable rule:comments +name: Private CA Desktop Release + +"on": + schedule: + - cron: "5 15 * * *" + workflow_dispatch: + issues: + types: [labeled] + +permissions: + contents: read + +concurrency: + group: private-ca-release-${{ github.event.issue.number || github.run_id }} + cancel-in-progress: false + +env: + UPSTREAM_REPOSITORY: block/buzz + PATCH_COMMIT: 6d03a38da5e3402bf97df1b3c46152887eb3778e + +jobs: + monitor: + if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + - name: Create one BUILD or SKIP ticket + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + release="$(gh api "repos/${UPSTREAM_REPOSITORY}/releases/latest")" + tag="$(jq -r '.tag_name' <<<"${release}")" + [[ "${tag}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "not a stable tag: ${tag}" >&2; exit 1; } + existing="$(gh issue list --state all --search "[Buzz update] ${tag} available in:title" --json number --jq '.[0].number // empty')" + [[ -z "${existing}" ]] || { echo "ticket already exists: #${existing}"; exit 0; } + work="$(mktemp -d)"; trap 'rm -rf "${work}"' EXIT + git clone --depth 1 --branch "${tag}" "https://github.com/${UPSTREAM_REPOSITORY}.git" "${work}/source" + source_sha="$(git -C "${work}/source" rev-parse HEAD)" + git -C "${work}/source" fetch --depth 2 "https://github.com/${GITHUB_REPOSITORY}.git" "${PATCH_COMMIT}" + if git -C "${work}/source" cherry-pick --no-commit "${PATCH_COMMIT}"; then + patch_status=clean + git -C "${work}/source" cherry-pick --abort || git -C "${work}/source" reset --hard + else + patch_status=conflict + git -C "${work}/source" cherry-pick --abort || git -C "${work}/source" reset --hard + fi + for label in buzz-update build-approved skip remediation-required built accepted; do + gh label create "${label}" --force --color 0E8A16 --description "Buzz private-CA release lifecycle" >/dev/null + done + cat >"${work}/issue.md" < + ## Buzz ${tag} is available + + Upstream release: $(jq -r '.html_url' <<<"${release}") + Published: $(jq -r '.published_at' <<<"${release}") + Immutable source SHA: \`${source_sha}\` + Patch applicability: **${patch_status}** + + ### Changelog + + $(jq -r '.body // "No upstream release notes supplied."' <<<"${release}") + + ### Decision required + + - Add \`build-approved\` to start the guarded arm64 build. Only @BrianInAz can approve. + - Add \`skip\` and close this issue to record no build is wanted. + - If applicability is \`conflict\`, use \`remediation-required\`; do not approve. + EOF + labels=(--label buzz-update) + if [[ "${patch_status}" == conflict ]]; then + labels+=(--label remediation-required) + fi + gh issue create --title "[Buzz update] ${tag} available" "${labels[@]}" --body-file "${work}/issue.md" + + skip: + if: >- + github.event_name == 'issues' && github.event.action == 'labeled' && + github.event.label.name == 'skip' && github.actor == 'BrianInAz' + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + steps: + - name: Record the explicit no-build decision + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + gh issue comment "${{ github.event.issue.number }}" --body "Skipped by @BrianInAz; no private-CA package was built." + gh issue close "${{ github.event.issue.number }}" + + approval: + if: >- + github.event_name == 'issues' && github.event.action == 'labeled' && + github.event.label.name == 'build-approved' && github.actor == 'BrianInAz' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: read + outputs: + issue_number: ${{ steps.verify.outputs.issue_number }} + tag: ${{ steps.verify.outputs.tag }} + source_sha: ${{ steps.verify.outputs.source_sha }} + steps: + - id: verify + env: + GH_TOKEN: ${{ github.token }} + ISSUE_NUMBER: ${{ github.event.issue.number }} + run: | + set -euo pipefail + body="$(gh issue view "${ISSUE_NUMBER}" --json body --jq .body)" + marker="$(grep -E '^$' <<<"${body}")" + [[ -n "${marker}" ]] || { echo "not a clean monitor-created ticket" >&2; exit 1; } + tag="$(sed -E 's/.*tag=([^ ]+).*/\1/' <<<"${marker}")" + source_sha="$(sed -E 's/.*source_sha=([^ ]+).*/\1/' <<<"${marker}")" + patch_sha="$(sed -E 's/.*patch_sha=([^ ]+).*/\1/' <<<"${marker}")" + [[ "${patch_sha}" == "${PATCH_COMMIT}" ]] || { echo "wrong patch" >&2; exit 1; } + remote_sha="$(git ls-remote "https://github.com/${UPSTREAM_REPOSITORY}.git" "refs/tags/${tag}^{}" | awk '{print $1}')" + [[ -n "${remote_sha}" ]] || remote_sha="$(git ls-remote "https://github.com/${UPSTREAM_REPOSITORY}.git" "refs/tags/${tag}" | awk '{print $1}')" + [[ "${remote_sha}" == "${source_sha}" ]] || { echo "tag SHA changed" >&2; exit 1; } + echo "issue_number=${ISSUE_NUMBER}" >> "$GITHUB_OUTPUT" + echo "tag=${tag}" >> "$GITHUB_OUTPUT" + echo "source_sha=${source_sha}" >> "$GITHUB_OUTPUT" + + validate: + needs: approval + runs-on: ubuntu-latest + timeout-minutes: 60 + permissions: + contents: read + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + - uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + - name: Fetch immutable source and apply patch + run: | + set -euo pipefail + git clone --depth 1 --branch "${{ needs.approval.outputs.tag }}" "https://github.com/${UPSTREAM_REPOSITORY}.git" "$RUNNER_TEMP/source" + [[ "$(git -C "$RUNNER_TEMP/source" rev-parse HEAD)" == "${{ needs.approval.outputs.source_sha }}" ]] + git -C "$RUNNER_TEMP/source" fetch --depth 2 "https://github.com/${GITHUB_REPOSITORY}.git" "${PATCH_COMMIT}" + git -C "$RUNNER_TEMP/source" cherry-pick --no-commit "${PATCH_COMMIT}" + git -C "$RUNNER_TEMP/source" diff --check + - name: Run full upstream CI + working-directory: ${{ runner.temp }}/source + run: just ci + - name: Run focused native connector test + working-directory: ${{ runner.temp }}/source + run: cargo test --manifest-path desktop/src-tauri/Cargo.toml native_websocket::tests::native_websocket_platform_tls_connector_is_available + + package: + needs: [approval, validate] + runs-on: macos-15 + timeout-minutes: 75 + permissions: + contents: write + attestations: write + id-token: write + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + - uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + - name: Fetch immutable source and apply patch + run: | + set -euo pipefail + git clone --depth 1 --branch "${{ needs.approval.outputs.tag }}" "https://github.com/${UPSTREAM_REPOSITORY}.git" "$RUNNER_TEMP/source" + [[ "$(git -C "$RUNNER_TEMP/source" rev-parse HEAD)" == "${{ needs.approval.outputs.source_sha }}" ]] + git -C "$RUNNER_TEMP/source" fetch --depth 2 "https://github.com/${GITHUB_REPOSITORY}.git" "${PATCH_COMMIT}" + git -C "$RUNNER_TEMP/source" cherry-pick --no-commit "${PATCH_COMMIT}" + - name: Run macOS platform connector test + working-directory: ${{ runner.temp }}/source + run: cargo test --manifest-path desktop/src-tauri/Cargo.toml native_websocket::tests::native_websocket_platform_tls_connector_is_available + - name: Install desktop dependencies + working-directory: ${{ runner.temp }}/source + run: just desktop-install-ci + - name: Build without updater artifacts or upstream signing + working-directory: ${{ runner.temp }}/source + run: | + set -euo pipefail + cat > desktop/src-tauri/tauri.private-ca.conf.json <<'EOF' + { "bundle": { "macOS": { "minimumSystemVersion": "10.15" }, "createUpdaterArtifacts": false } } + EOF + cd desktop + pnpm tauri build --verbose --no-sign --features mesh-llm --config src-tauri/tauri.private-ca.conf.json + - name: Ad-hoc sign and rebuild the arm64 DMG + working-directory: ${{ runner.temp }}/source + run: | + set -euo pipefail + app_path="desktop/src-tauri/target/release/bundle/macos/Buzz.app" + mkdir -p "$RUNNER_TEMP/release/dmg-root" + codesign --force --deep --sign - "${app_path}" + codesign --verify --deep --strict --verbose=2 "${app_path}" + cp -R "${app_path}" "$RUNNER_TEMP/release/dmg-root/Buzz.app" + hdiutil create -volname Buzz -srcfolder "$RUNNER_TEMP/release/dmg-root" -ov -format UDZO "$RUNNER_TEMP/release/Buzz-${{ needs.approval.outputs.tag }}-private-ca-arm64.dmg" + - name: Create manifest, patch, and checksums + working-directory: ${{ runner.temp }}/source + run: | + set -euo pipefail + cat > "$RUNNER_TEMP/release/manifest.json" < "$RUNNER_TEMP/release/private-ca.patch" + (cd "$RUNNER_TEMP/release" && shasum -a 256 *.dmg manifest.json private-ca.patch > SHA256SUMS) + printf '%s\n' '# Private-CA build test summary' '- upstream tag and SHA verified' '- just ci and focused connector test passed on Ubuntu' '- macOS platform connector test passed' '- ad-hoc signed DMG with updater disabled' > "$RUNNER_TEMP/release/TEST-SUMMARY.md" + - name: Attest the DMG + uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1 + with: + subject-path: ${{ runner.temp }}/release/*.dmg + - name: Publish fork prerelease and mark ticket built + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + release_tag="buzz-private-ca-${{ needs.approval.outputs.tag }}-r${GITHUB_RUN_NUMBER}" + gh release create "${release_tag}" "$RUNNER_TEMP/release/Buzz-${{ needs.approval.outputs.tag }}-private-ca-arm64.dmg" "$RUNNER_TEMP/release/SHA256SUMS" "$RUNNER_TEMP/release/manifest.json" "$RUNNER_TEMP/release/private-ca.patch" "$RUNNER_TEMP/release/TEST-SUMMARY.md" --prerelease --title "Buzz private-CA ${release_tag}" --notes "Built from ${{ needs.approval.outputs.tag }} at ${{ needs.approval.outputs.source_sha }} with patch ${PATCH_COMMIT}." + gh issue edit "${{ needs.approval.outputs.issue_number }}" --add-label built + gh issue comment "${{ needs.approval.outputs.issue_number }}" --body "Fork prerelease published: https://github.com/${GITHUB_REPOSITORY}/releases/tag/${release_tag}" diff --git a/scripts/test-private-ca-release-contract.sh b/scripts/test-private-ca-release-contract.sh new file mode 100755 index 0000000000..4b76379aeb --- /dev/null +++ b/scripts/test-private-ca-release-contract.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash + +set -euo pipefail + +workflow=".github/workflows/private-ca-release.yml" + +if [[ ! -f "${workflow}" ]]; then + echo "missing ${workflow}" >&2 + exit 1 +fi + +require() { + local expected="$1" + if ! rg --fixed-strings --quiet -- "${expected}" "${workflow}"; then + echo "${workflow} must contain: ${expected}" >&2 + exit 1 + fi +} + +forbid() { + local prohibited="$1" + if rg --ignore-case --fixed-strings --quiet -- "${prohibited}" "${workflow}"; then + echo "${workflow} must not contain: ${prohibited}" >&2 + exit 1 + fi +} + +require 'cron: "5 15 * * *"' +require 'issues:' +require 'labeled' +require 'workflow_dispatch:' +require 'contents: read' +require 'issues: write' +require 'attestations: write' +require 'id-token: write' +require "github.actor == 'BrianInAz'" +require "github.event.label.name == 'build-approved'" +require "github.event.label.name == 'skip'" +require 'refs/tags/' +require '6d03a38da5e3402bf97df1b3c46152887eb3778e' +require 'cherry-pick --no-commit' +require 'just ci' +require 'macos-15' +require 'codesign --verify --deep --strict' +require 'hdiutil create' +require 'SHA256SUMS' +require 'gh release create' +require 'gh issue close' +require '--prerelease' +require 'actions/attest-build-provenance@' +require 'createUpdaterArtifacts": false' +require 'buzz-private-ca-' + +forbid 'BUZZ_TEST_WSS_URL' +forbid 'buzz.bjzy.me' +forbid 'insecure_skip_verify' +forbid 'tailscale' +forbid 'VAULT_' +forbid 'APPLE_CERTIFICATE' +forbid 'TAURI_SIGNING_PRIVATE_KEY' + +echo "private CA release workflow contract passed"