diff --git a/CHANGELOG.md b/CHANGELOG.md index 6801fc626..d65e9d16e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,46 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [1.15.0] - 2026-08-17 + +Minor release on attachments and external tools. PowerPoint decks can now be uploaded and handed to the PowerPoint toolset, and attachment cards survive a reload โ€” a gap that also affected spreadsheets. OAuth-gated MCP servers no longer lose their tools permanently when the pre-flight runs on a cold token cache, and an abandoned consent prompt no longer bricks every later message in the conversation. Two IAM grants missing in production are fixed, and ALB access logs are enabled so a mid-stream disconnect can be attributed to whoever actually ended the connection. **Requires a CDK deploy** โ€” run `platform.yml`, then `backend.yml`, then `frontend-deploy.yml`. + +### ๐Ÿš€ Added + +- `.pptx` uploads accepted in the SPA and routed to the PowerPoint toolset, with a presentation carve-out that keeps decks out of the inline document set Bedrock cannot encode (#868) +- Slide-deck preview card for `.pptx` attachments โ€” presentation icon and a stacked-slides mock in place of the generic grey file chip (#873) +- `navigated_away` interruption reason, attested by a SPA `pagehide` handler so refreshes, tab closes, and navigations leave the unattributable `connection_lost` bucket (#864) +- ALB access logs to a 30-day SSE-S3 bucket, recording `elb_status_code`, termination reason, and `request_processing_time` for connection-termination attribution (#867) + +### ๐Ÿ› Fixed + +- OAuth-gated MCP tools were dropped permanently when the registration pre-flight 401'd against a cold `oauth_token_cache`; the runtime now consults the AgentCore vault, warms the cache and retries, or records a consent gap instead of dropping the tool silently. 20 turns hit this in prod in 24h (#872) +- An unreachable MCP server was misread as an OAuth consent gap, showing a Connect prompt that completing consent could never satisfy; only 401/403 now counts, with the status read through the wrapped `ToolProviderException` โ†’ `MCPClientInitializationError` โ†’ `ExceptionGroup` chain (#876) +- An abandoned OAuth-consent or tool-approval pause left `InterruptState.activated` set on the cached agent, so every later turn in the session failed with a non-recoverable `stream_error`; a fresh turn now abandons the stale pause and drops history back to the last completed assistant turn (#874) +- `.pptx` and spreadsheet attachment cards vanished on reload โ€” carved-out files never reached the `[Attached files: โ€ฆ]` marker the SPA rebuilds them from, and a lone deck left no trace at all (#873) +- app-api was missing `s3vectors:DeleteVectors`, so document cleanup exhausted its retries on every vector delete and orphaned chunks in the index until TTL โ€” two bulk cleanups failed outright in one hour (#870) +- The AgentCore runtime role lacked read access to the user-settings table, so `get_settings` swallowed the `AccessDenied` into `DEFAULT_SETTINGS` and silently ignored each user's saved `defaultModelId` (#870) +- `validateFile` accepted a file by extension when the browser reported no MIME, but `uploadFile` then sent `application/octet-stream`, 400-ing an upload the UI had just accepted; `resolveMimeType()` now resolves from the extension (#868) + +### โš ๏ธ Changed + +- SSE contract: `oauth_required.interruptId` is now optional. The pre-flight flavor has no paused turn to resume โ€” clients must show the Connect affordance without attempting a resume (#872) +- `build_prompt` takes a new `attachment_names` argument, defaulting to the names in `files` so existing callers are unchanged (#873) + +### ๐Ÿ—๏ธ Infrastructure + +- New ALB access-log S3 bucket (SSE-S3, 30-day expiry). SSE-KMS is not an option โ€” the ELB log-delivery service fails silently against a KMS-encrypted bucket (#867) +- New app-api env var `FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION` (25MB), a deck-specific ceiling above the general 4MB inline-document cap (#868) +- Synthing the ALB construct now requires a concrete region, since CDK resolves the regional ELB log-delivery principal for the bucket policy (#867) +- `S3VectorsQueryAccess` gains `s3vectors:DeleteVectors`; new `UserSettingsTableReadAccess` on the AgentCore runtime role (#870) + +### ๐Ÿ“š Docs + +- AgentCore Evaluations spike findings and eval-harness scope (#862) +- Expanded Bedrock Managed KB evaluation with recommendations (#867) +- G3 document-citations probe findings โ€” the premise was wrong; visual PDF understanding is unconditional and citations are text-layer-only (#869) +- Weekly kaizen research scan and review prep for 2026-08-14, plus nine stale review-queue entries resolved (#871, #875) + ## [1.14.1] - 2026-08-13 Patch release on the streaming interrupt path. A dropped SSE connection no longer bricks the next message in the conversation โ€” the fix users are most likely to notice, since the failure presented as a "Network error" followed by a fatal "Chat Request Failed" on the resend. Stop now reaches an in-flight MCP call instead of waiting it out, and a cancelled turn no longer sticks to the cached agent and refuses every subsequent message. Session persistence moves off the asyncio event loop, and the Strands/AgentCore libraries move to the versions those fixes require. **No CDK deploy** โ€” `backend.yml`, then `frontend-deploy.yml`. diff --git a/CLAUDE.MD b/CLAUDE.MD index 90c8a47c1..14dc97705 100644 --- a/CLAUDE.MD +++ b/CLAUDE.MD @@ -58,7 +58,7 @@ npx cdk deploy {prefix}-PlatformStack | `session_title` | Server-generated conversation title on a session's FIRST turn โ€” payload `{type, sessionId, title}`. Title generation (Nova Micro) runs as an asyncio task concurrent with the agent stream; the finished title is interleaved between agent events (non-blocking done-check in `stream_with_quota_warning`), so the sidebar/top-nav rename while the response is still pending. Emitted at most once per stream, possibly after `done` (the SPA parser allowlists it past Completed-state gating); never carries the "New Conversation" placeholder. Best-effort: a stream that finishes before generation emits nothing โ€” the SPA's post-close metadata refresh (`refreshTitleFromServer`) is the fallback, reading the title the task also persisted via `update_session_title` | | `quota_session_notice` | **This conversation** has reached the tier's session-notice share of the monthly limit โ€” payload `{type, sessionId, sessionCost, quotaLimit, sessionPercentageOfLimit, thresholdPercentage, message}`. Emitted at the head of the stream right after `quota_warning`, and re-emitted every turn while over the share (dismissal is client-side, same contract as `quota_warning`). `sessionCost` is the session's **lifetime** cost โ€” the `totalCost` aggregate on its metadata row โ€” deliberately not period-scoped: a conversation that opened last month and is spending this month's budget is exactly the one worth surfacing. Share is tier-configurable (`sessionNoticePercentage`, default 25%, 0 disables); the whole runway rides the `QUOTA_RUNWAY_ENABLED` kill switch (default on), which also gates the 50%/75% `quota_warning` rungs. The SPA scopes it to the conversation it names โ€” never shown above another thread's composer | | `stream_error` | Conversational error | -| `oauth_required` | External MCP tool needs user consent โ€” payload `{providerId, authorizationUrl}`, one event per provider emitted after `message_stop` | +| `oauth_required` | External MCP tool needs user consent โ€” payload `{providerId, authorizationUrl, interruptId?}`, one event per provider emitted after `message_stop`. Two flavors. **Interrupt-driven** (`interruptId` present): `OAuthConsentHook` paused a tool call mid-turn; the SPA resumes that exact turn by POSTing the id back. **Pre-flight** (`interruptId` *absent*): the tool never registered because the MCP server refused the pre-flight `tools/list` โ€” since the consent hook is `BeforeToolCall`, it can't fire for an unregistered tool, so without this event the tool vanishes with no explanation. Nothing is paused, so the SPA shows the Connect affordance and must NOT resume; a synthetic id would be worse than none, because the resume guard in `inference_api/chat/routes.py` 400s on unknown ids and the user would hit an error right after consenting. Pre-flight events are re-emitted each turn that rebuilds the agent (the pre-flight keeps failing until consent lands) and are deliberately not persisted as `pending_interrupt` breadcrumbs โ€” those are keyed by interrupt id for the resume path. The SPA dedupes by `providerId` and suppresses a dismissed pre-flight prompt for the tab session. Before emitting, the runtime asks the AgentCore vault directly: a vaulted token is warmed into `oauth_token_cache` and the pre-flight retried, so a user who already consented gets the tool back instead of losing it for the life of the process | | `compaction` | Backend rolled older turns into a summary on this turn โ€” payload `{previousCheckpoint, newCheckpoint, summarizedTurns, inputTokens}`, emitted after the final `metadata` event so the badge updates first, before `done` | | `done` | Stream complete | diff --git a/README.md b/README.md index 13d16baff..647e31a34 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **An open-source, production-ready Generative AI platform for institutions** *Built by Boise State University, designed for everyone.* -[![Release](https://img.shields.io/badge/Release-v1.14.1-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) +[![Release](https://img.shields.io/badge/Release-v1.15.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml) ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white) @@ -296,7 +296,7 @@ agentcore-public-stack/ See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release. -**Current release:** v1.14.1 +**Current release:** v1.15.0 --- diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 9456c7f03..55fb1736a 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,176 @@ +# Release Notes โ€” v1.15.0 + +**Release Date:** August 17, 2026 +**Previous Release:** v1.14.1 (August 13, 2026) + +--- + +> ๐Ÿ—๏ธ **CDK deploy required.** Run `platform.yml` first, then `backend.yml`, then `frontend-deploy.yml`. `infrastructure/lib/` changed in four places this release (one new S3 bucket, two IAM grants, one env var). `infrastructure/gsi-inventory.json` is byte-identical to `main` โ€” **no data migration, no GSI changes**. + +--- + +## Highlights + +A minor release about attachments and external tools. + +**PowerPoint decks are uploadable.** The backend has accepted `.pptx` since the PowerPoint toolset landed, but the SPA's allowlist never did โ€” so `create_powerpoint_presentation`'s own error text told users to "upload a .pptx template first," advice the UI made impossible. Decks now upload, get a stacked-slides preview card instead of a grey blob, and route to the PowerPoint tools. Fixing the reload path for them fixed it for spreadsheets too: any attachment that skips the inline document set was vanishing from the conversation on refresh. + +**OAuth-gated MCP servers stop losing their tools.** A server that requires auth even for `tools/list` โ€” GitHub's does โ€” 401'd the registration pre-flight on every fresh microVM, and the tool was dropped with no explanation and no recovery for the life of the process. That was the single largest source of ERROR lines in the production runtime log. Alongside it, an abandoned consent prompt no longer bricks every later message in the conversation. + +**Two IAM grants missing in production are fixed** โ€” both had the capability wired end-to-end except the policy statement, so neither surfaced as a user-visible error while document cleanup silently orphaned vector chunks and every user's saved default model was silently ignored. + +--- + +## PowerPoint decks as attachments + +Users can upload a `.pptx` and hand it to the PowerPoint tools โ€” as a template to build from, or a deck to read. It renders as a slide-deck card in the conversation and survives a reload. + +The upload could not simply be allowlisted. Bedrock's Converse `DocumentFormat` enum has no `pptx` member (`pdf`, `csv`, `doc`, `docx`, `xls`, `xlsx`, `html`, `txt`, `md`), so a deck sent as an inline document block fails the turn with a `ValidationException` at any size. The release adds a presentation carve-out mirroring the existing tabular one, diverting decks out of the inline set and pointing the model at the tools instead. + +### Backend + +- `apis/shared/files/models.py` โ€” new `is_presentation_file()` predicate +- `_partition_attachments()` returns a 4-tuple, diverting decks **before** the size gate (they never go inline, so an "oversized" note would misdescribe why they were skipped) +- `_build_attachment_guidance()` names the deck and points at `read_powerpoint_presentation`, or names the toggle when the tool is disabled +- `PromptBuilder.build_prompt()` takes an authoritative `attachment_names` list, defaulting to the names in `files` so existing callers are unchanged. Its `if not files: return message` early path now still emits the marker +- `chat_agent.stream_async` forwards it; the route derives it via `_attachment_marker_names`, the only place that sees every attachment. Oversized files stay excluded โ€” those were dropped from the turn and the guidance already explains their absence +- The tools needed no changes: `_find_powerpoint_presentation` already resolves any `READY` `.pptx` in the session +- Decks get their own **25MB** upload ceiling (`FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION`) rather than the general 4MB cap, since that cap exists to bound inline document blocks and a deck never becomes one + +### Frontend + +- `file-upload.service.ts` โ€” `.pptx` added to the allowlist; new `resolveMimeType()` resolves from the extension. `validateFile` accepted a file by extension when the browser reported no MIME, but `uploadFile` then sent `application/octet-stream`, which the backend allowlist rejects โ€” 400-ing an upload the UI had just accepted. It bit decks twice, since `_find_powerpoint_presentation` matches the stored MIME exactly, so a deck saved as octet-stream would upload and then be invisible to the tool +- `file-attachment-badge.component.ts` โ€” `FILE_TYPE_STYLES` gains a PPTX entry (presentation icon, orange tint). Presentations render a 16:9 mock front slide with two offset slides stacked behind it, and suppress the folded-corner and bottom-fade details โ€” both are "sheet of paper" cues that fight the stacked-slides metaphor. The mock slide is decorative; a real first-slide thumbnail needs LibreOffice + +### Why attachment cards vanished on reload + +The card renders from a `fileAttachment` content block the SPA builds client-side at send time and never persists. On reload it rebuilds those blocks by parsing the `[Attached files: โ€ฆ]` marker out of the message text and matching names against the session's file list (`restoreFileAttachments` in `message-map.service.ts`) โ€” so that marker is the only surviving link between a file and the message it was attached to. `PromptBuilder` derived the marker from the files it was turning into content blocks, i.e. the inline set, so any carved-out file never reached it. Spreadsheets had the same gap from the tabular carve-out; csv/xlsx cards now survive reload too. + +### Test Coverage + +420+ lines across `test_presentation_attachment_carveout.py`, `test_presentation_upload_size_cap.py`, and `test_attachment_marker.py`, plus a frontend spec that pins `FILE_TYPE_STYLES` against the upload allowlist โ€” a type missing from the map degrades silently to a grey blob rather than failing, which is exactly how this shipped, so adding a type to one list now forces the other. + +--- + +## OAuth-gated MCP tools survive a cold token cache + +An MCP server that requires auth even for `tools/list` 401s the registration pre-flight whenever the in-process `oauth_token_cache` is cold โ€” which it always is on a fresh microVM. `load_external_tools` caught that, logged a warning, and dropped the tool. + +Nothing recovered from there. `OAuthConsentHook` is a `BeforeToolCall` hook, so it only runs for tools that made it into the registry; the dropped tool never reached it, the cache was never warmed, and the drop repeated every turn for the life of the process. A user whose token was sitting in the AgentCore vault the whole time lost the tool permanently and was told nothing โ€” the model simply did not have it. **20 turns hit this in production in 24 hours, the single largest source of ERROR lines in the runtime log.** + +### Backend + +On pre-flight failure for an OAuth-gated tool with a cold cache, the runtime now asks the vault directly: + +- **token** โ†’ warm the cache and retry the pre-flight once. This is the consented-user path, and it is also how the tool returns by itself after the user completes consent in the popup +- **consent URL** โ†’ AgentCore is saying the user genuinely has not authorized. Record it so the turn emits `oauth_required` rather than dropping the tool with no explanation +- **hard error** โ†’ stay silent. "Couldn't ask" is not "must consent"; prompting there would nag a connected user whenever the server blips + +The vault is only consulted when the pre-flight already failed, so the happy path costs no extra round-trip. Scopes and `customParameters` moved into a shared `oauth/token_resolution.py` helper โ€” AgentCore folds both into the token-vault key, so two callers asking with different values look up different vault entries and would prompt a user who is already connected. + +A follow-up narrowed the trigger. Recovery originally ran on *any* pre-flight failure, so an unreachable server looked identical to one refusing an unauthorized caller: the vault correctly answered "this user has no token, here is an authorization URL," and the user got a Connect prompt on every turn that completing consent could never satisfy. Only a 401/403 now counts. The status is not on the exception that surfaces โ€” `MCPClient.load_tools()` raises `ToolProviderException` wrapping `MCPClientInitializationError` wrapping an anyio `ExceptionGroup`, with the real `httpx.HTTPStatusError` three levels down and no status in the outermost message โ€” so `_is_auth_failure` walks `__cause__`/`__context__`/`ExceptionGroup` members, the same shape `mcp_apps._is_transient_connect_error` already had to handle. + +### SSE contract + +`oauth_required.interruptId` **is now optional.** The pre-flight flavor has no paused turn to resume, and a synthetic id would be worse than none โ€” the resume guard in `inference_api/chat/routes.py` 400s on unknown ids, so the user would hit an error immediately after consenting. Clients must show the Connect affordance and not attempt a resume when the field is absent. Pre-flight events are re-emitted each turn that rebuilds the agent and are deliberately not persisted as `pending_interrupt` breadcrumbs, which are keyed by interrupt id for the resume path. + +### Frontend + +- `oauth-consent.service.ts` and `stream-parser-core.ts` โ€” handle the id-less flavor, dedupe by `providerId`, and suppress a dismissed pre-flight prompt for the tab session + +### Test Coverage + +680+ lines across `test_external_mcp_client.py`, `test_oauth_token_resolution.py`, and `test_preflight_consent_events.py`. The tests build the real wrapped exception chain; an existing test that used `RuntimeError("connection refused")` to reach the vault-unreachable branch now uses a 401, since under the new gate it would never have reached the vault and would have passed vacuously. + +--- + +## Attributing interrupted turns + +`connection_lost` is a fallback label, not a diagnosis. The container stamps it whenever a stream task is torn down with no client signal, so a refresh, a dead socket, and a platform-side idle timeout were all recorded identically and the resulting population could not be reasoned about. Over 14 days in production: 64 `connection_lost` against 40 `user_stopped`, and of 47 dropped turns sampled in detail only the 11 in the 62โ€“67s band could be attributed to anything at all. The rest were unexplainable in principle, not merely unexplained. + +This release closes both ends of that gap โ€” the browser's and the load balancer's. + +### Frontend + +A page departure is the one interruption cause only the browser witnesses, and the SPA had no `beforeunload`/`pagehide`/`unload` handler anywhere โ€” `cancelChatRequest` was reached solely from the Stop button. A `pagehide` handler now signals `navigated_away` for each session with a stream in flight. `pagehide` rather than `unload` because it still fires for mobile Safari and bfcache navigations; `keepalive: true` on the fetch is what lets the request outlive the page, the same property the Stop path already relies on. + +Two things it deliberately does **not** do: + +- **It does not abort the stream.** Aborting on page-hide would kill turns for a bfcache navigation the user may return from, and the server turn is meant to keep running so a reload can offer to continue it +- **It does not arm the distributed turn cancel.** That stays exclusive to a deliberate Stop โ€” cancelling on a departure would make every refresh discard work the reload is about to offer to continue. This endpoint's reason set widened; its side effects did not + +### Infrastructure + +The ALB access log is the only record of who *ended* a connection. A mid-stream SSE disconnect is indistinguishable from inside the container โ€” client gone, socket dropped, and the ALB's own 60s idle timeout all arrive as the same cancellation โ€” so with logging off, attribution beyond the timing signature was guesswork. `elb_status_code`, the termination-reason field, and `request_processing_time` name the terminator and how long the request had run. + +- New access-log bucket is **SSE-S3, not KMS**: the ELB log-delivery service cannot write to an SSE-KMS bucket and fails silently, leaving an empty bucket and no logs. 30-day expiry bounds the cost of a per-request log +- Synthing the ALB construct now requires a concrete region โ€” CDK resolves the regional ELB log-delivery principal for the bucket policy and refuses on an env-agnostic stack. Every real deploy already passes env via `bin/infrastructure.ts`; one `PlatformStack` test did not, and now does + +--- + +## ๐Ÿ› Bug fixes + +**An abandoned consent prompt bricked every later message.** If a turn paused on an OAuth-consent or tool-approval interrupt and the user never completed it โ€” just typed a new message instead โ€” Strands rejected the new turn with `TypeError: prompt_type= | must resume from interrupt with list of interruptResponse's`. `InterruptState.resume` refuses a plain string prompt while `activated` is set, and that flag lives on the agent, which the cache reuses across turns. Nothing cleared it, so every later turn in the session hit the same wall and produced a non-recoverable `stream_error`. + +The "a fresh turn supersedes a paused turn" policy already existed in `clear_paused_turn` / `clear_interrupted_turn`, but it only cleared the DynamoDB side โ€” the live object on the cached agent was missed. Deactivating alone is not enough: Strands appends the assistant `toolUse` before running tools and returns on interrupt without the matching `toolResult`, so history ends on an unanswered tool call. `_repair_tool_pairing` cannot fix it, because it deliberately leaves a *trailing* `toolUse` alone for prompt-arrival handling and does not count it as a violation. So the abandoned turn is dropped back to the last completed assistant turn, in place โ€” the message list is aliased across the cached agents serving one session, and rebinding mid-life silently breaks that alias (#874) + +**Document cleanup orphaned vector chunks in production.** app-api's `S3VectorsQueryAccess` listed read actions only, so `documents/services/cleanup_service.py` exhausted its three retries on every vector delete and logged "Cleanup incomplete โ€ฆ TTL will auto-expire". Two bulk cleanups failed outright in a single hour (0/18 and 0/1 documents), leaving those chunks searchable in the index until TTL. Note the batch action is `DeleteVectors` (plural); rag-ingestion's `DeleteVector` (singular) is a different action (#870) + +**Saved default model was silently ignored.** `inference-agentcore-construct.ts` injects `DYNAMODB_USER_SETTINGS_TABLE_NAME`, so `UserSettingsRepository` reported itself enabled โ€” but the table was absent from the runtime role's grants, and `get_settings` swallowed the `AccessDenied` into `DEFAULT_SETTINGS`, serving the system default instead of the user's `defaultModelId`. Scoped read-only on a bare ARN following the `SystemPromptsTableReadAccess` precedent; the runtime never writes settings and the table has no GSIs (#870) + +Both IAM gaps were found by `AccessDeniedException` in the prod-ai CloudWatch logs, and both now carry regression tests alongside the existing `SharedConversationsAccess` guard, which covers the identical failure mode. + +--- + +## ๐Ÿ—๏ธ Infrastructure + +| Change | Detail | +|---|---| +| ALB access-log bucket | New S3 bucket, SSE-S3 encryption, 30-day expiry. KMS is not an option โ€” ELB log delivery fails silently against it | +| `S3VectorsQueryAccess` | Gains `s3vectors:DeleteVectors` on the app-api task role | +| `UserSettingsTableReadAccess` | New read-only grant on the AgentCore runtime role | +| `FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION` | New app-api env var, 25MB. Must never be smaller than `PPTX_MAX_FILE_SIZE_BYTES` in the SPA's `file-upload.service.ts` | +| Synth requirement | The ALB construct now needs a concrete region for the regional ELB log-delivery principal | + +No GSI changes: `infrastructure/gsi-inventory.json` is byte-identical to `main`, and `scripts/release/check-gsi-update-limit.mjs` passes across all 26 tables. No data migration. + +--- + +## ๐Ÿ“š Documentation + +- **G3 document-citations probe** โ€” the premise was wrong. Visual PDF understanding is unconditional, and citations are text-layer-only rather than the gate we assumed. Note the answer text moves inside `citationsContent` (#869) +- **AgentCore Evaluations spike** and eval-harness scoping (#862) +- **Bedrock Managed KB evaluation** expanded with recommendations (#867) +- Weekly kaizen research scan and review prep for 2026-08-14, with nine stale review-queue entries resolved (#871, #875) + +--- + +## ๐Ÿงช Test coverage + +**2,600+ lines of new tests.** Largest additions: `test_external_mcp_client.py` (399), `test_stale_interrupt_reset.py` (273), `test_presentation_attachment_carveout.py` (194), `test_oauth_token_resolution.py` (165), `test_preflight_consent_events.py` (120), `test_presentation_upload_size_cap.py` (117), `test_attachment_marker.py` (109). Infrastructure adds 90 lines of security-policy assertions covering both new IAM grants and 48 lines for the access-log bucket. + +--- + +## ๐Ÿš€ Deployment notes + +**This release requires a CDK deploy**, unlike 1.14.1. + +1. **`platform.yml`** โ€” picks up the ALB access-log bucket, both IAM grants, and the new app-api env var +2. **`backend.yml`** โ€” app-api, inference-api, rag-ingestion, artifact-render +3. **`frontend-deploy.yml`** โ€” S3 + CloudFront + +No data migration and no GSI changes, so the deploy is not order-sensitive beyond the sequence above. Compute image URIs still come from SSM at CFN deploy time, so the infra deploy will not revert a live service. + +**After deploying, verify:** + +- The ALB access-log bucket is receiving objects. An empty bucket after real traffic means the log-delivery grant did not take โ€” check the bucket is SSE-S3 and not KMS +- A document delete no longer logs "Cleanup incomplete โ€ฆ TTL will auto-expire" in the app-api logs +- A user with a non-default `defaultModelId` gets that model on a fresh session + +**For API clients:** `oauth_required.interruptId` is now optional. Any client that resumes a turn from that event must check for the field's presence and fall back to showing a Connect affordance without a resume โ€” posting a missing or synthetic id back will 400. + +--- + # Release Notes โ€” v1.14.1 **Release Date:** August 13, 2026 diff --git a/VERSION b/VERSION index 63e799cf4..141f2e805 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.14.1 +1.15.0 diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 6ce6f2e7d..610bd4364 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agentcore-stack" -version = "1.14.1" +version = "1.15.0" requires-python = ">=3.10" description = "Multi-agent conversational AI system with AWS Bedrock AgentCore" readme = "README.md" diff --git a/backend/scripts/probe_document_citations.py b/backend/scripts/probe_document_citations.py new file mode 100644 index 000000000..b8240ee49 --- /dev/null +++ b/backend/scripts/probe_document_citations.py @@ -0,0 +1,423 @@ +"""G3 baseline probe โ€” what does today's document path actually see? + +`docs/specs/document-offload-evaluation.md` ยง1 requires this before any offload +quality scoring: the validation pass found we send no citations config in +production, and assumed that on Bedrock the visual (page-image) PDF path is +tied to citations-enabled document handling. If that were true, production +would be blind to charts today and every offload comparison would inherit a +degraded baseline. + +This probe settles it. It builds a self-contained corpus, then asks each +question twice against Bedrock Converse: + + arm "bare" โ€” the document block exactly as `DocumentHandler` builds it + today (format / name / source.bytes, no citations) + arm "cited" โ€” the same block with `citations: {enabled: True}` added, + nothing else changed + +Four of the five documents are image-only (PIL writes no text layer), so a +correct answer proves the model saw pixels. `text_layer.pdf` is the probe's +own canary: if it fails, the probe is broken rather than the model. The mixed +document is the realistic production shape โ€” prose plus a figure. + +Usage: + cd backend + AWS_PROFILE=dev-ai uv run python scripts/probe_document_citations.py + AWS_PROFILE=dev-ai uv run python scripts/probe_document_citations.py \ + us.anthropic.claude-haiku-4-5-20251001-v1:0 us.anthropic.claude-sonnet-5 + +Findings as of 2026-08-12 are recorded in +`docs/specs/document-citations-probe-findings.md`. +""" + +from __future__ import annotations + +import argparse +import io +import json +import logging +import os +import re +import tempfile +import time +from typing import Any, Dict, List, Optional, Tuple + +import boto3 +from PIL import Image, ImageDraw, ImageFont + +logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") +logger = logging.getLogger("g3-probe") + +REGION = "us-west-2" +DEFAULT_MODELS = ["us.anthropic.claude-haiku-4-5-20251001-v1:0"] + +FONT_CANDIDATES = [ + "/System/Library/Fonts/Supplemental/Arial.ttf", + "/System/Library/Fonts/Helvetica.ttc", + "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", +] + +REFUSAL_MARKERS = [ + "unable to", "cannot see", "can't see", "no image", "not able to see", + "don't have access", "do not have access", "unable to view", "cannot view", + "not visible", "cannot read", "can't read", +] + + +# --------------------------------------------------------------------- corpus + +def _font(size: int) -> Any: + for path in FONT_CANDIDATES: + if os.path.exists(path): + try: + return ImageFont.truetype(path, size) + except OSError: + continue + return ImageFont.load_default() + + +def _chart_image() -> Image.Image: + """Bar chart whose values exist only as pixel heights against an axis.""" + width, height = 1200, 900 + img = Image.new("RGB", (width, height), "white") + draw = ImageDraw.Draw(img) + draw.text((60, 40), "Quarterly Enrollment, Department of Ceramics", + font=_font(34), fill="black") + + bars = [("Fall 2023", 412), ("Spring 2024", 268), ("Fall 2024", 631), ("Spring 2025", 349)] + base_y, left, bar_w, gap, top_val = 780, 140, 150, 90, 700 + scale = (base_y - 160) / top_val + + draw.line([(left - 40, base_y), (width - 60, base_y)], fill="black", width=3) + draw.line([(left - 40, base_y), (left - 40, 150)], fill="black", width=3) + for grid in range(0, top_val + 1, 100): + y = base_y - grid * scale + draw.line([(left - 48, y), (left - 40, y)], fill="black", width=3) + draw.text((left - 120, y - 14), str(grid), font=_font(24), fill="black") + + for i, (label, value) in enumerate(bars): + x0 = left + i * (bar_w + gap) + draw.rectangle([x0, base_y - value * scale, x0 + bar_w, base_y], fill=(41, 84, 143)) + draw.text((x0 + 6, base_y + 14), label, font=_font(22), fill="black") + return img + + +def _table_image() -> Image.Image: + width, height = 1200, 800 + img = Image.new("RGB", (width, height), "white") + draw = ImageDraw.Draw(img) + draw.text((60, 40), "Table 3. Course Fees by Program (2025-26)", font=_font(32), fill="black") + rows = [ + ["Program", "Lab Fee", "Materials", "Total"], + ["Ceramics", "$145", "$310", "$455"], + ["Printmaking", "$92", "$188", "$280"], + ["Metalsmithing", "$237", "$401", "$638"], + ["Photography", "$118", "$264", "$382"], + ] + x0, y0, col_w, row_h = 70, 130, 265, 76 + for r, row in enumerate(rows): + for c, cell in enumerate(row): + x, y = x0 + c * col_w, y0 + r * row_h + draw.rectangle([x, y, x + col_w, y + row_h], outline="black", width=2) + if r == 0: + draw.rectangle([x + 2, y + 2, x + col_w - 2, y + row_h - 2], fill=(230, 230, 235)) + draw.text((x + 16, y + 24), cell, font=_font(26), fill="black") + return img + + +def _scan_image() -> Image.Image: + width, height = 1200, 1000 + img = Image.new("RGB", (width, height), (252, 251, 246)) + draw = ImageDraw.Draw(img) + lines = [ + "MEMORANDUM", "", "To all department chairs:", "", + "Effective the start of the spring term, the equipment replacement", + "reserve will be held at eleven percent of each department's annual", + "operating allocation. Requests to draw against the reserve must be", + "filed with the facilities office no later than the fourteenth day of", + "the month preceding the intended purchase.", "", + "The prior threshold of six percent is retired and should not be used", + "in any budget projection after this date.", "", + " -- Office of the Provost, document reference PR-2291", + ] + y = 90 + for line in lines: + draw.text((100, y), line, font=_font(30), fill=(28, 28, 32)) + y += 56 + img = img.rotate(-0.7, expand=False, fillcolor=(252, 251, 246)) + ImageDraw.Draw(img).rectangle([0, 0, width - 1, height - 1], outline=(205, 203, 197), width=6) + return img + + +def _assemble_pdf(objects: Dict[int, bytes]) -> bytes: + """Serialize numbered PDF objects into a valid single-file PDF.""" + out = b"%PDF-1.4\n" + offsets: Dict[int, int] = {} + for num in sorted(objects): + offsets[num] = len(out) + out += f"{num} 0 obj\n".encode("latin-1") + objects[num] + b"\nendobj\n" + xref_at = len(out) + out += f"xref\n0 {len(objects)+1}\n0000000000 65535 f \n".encode("latin-1") + for num in sorted(objects): + out += f"{offsets[num]:010d} 00000 n \n".encode("latin-1") + out += (f"trailer\n<< /Size {len(objects)+1} /Root 1 0 R >>\n" + f"startxref\n{xref_at}\n%%EOF\n").encode("latin-1") + return out + + +def _text_layer_pdf() -> bytes: + body = ( + "BT /F1 16 Tf 60 720 Td (Facilities Standards Handbook, Section 9) Tj ET\n" + "BT /F1 12 Tf 60 690 Td (The freight elevator in the Liberal Arts building has a rated) Tj ET\n" + "BT /F1 12 Tf 60 672 Td (capacity of 3,400 pounds and is inspected twice per year.) Tj ET\n" + "BT /F1 12 Tf 60 654 Td (The passenger elevators are inspected annually.) Tj ET\n" + ) + return _assemble_pdf({ + 1: b"<< /Type /Catalog /Pages 2 0 R >>", + 2: b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", + 3: b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] " + b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>", + 4: f"<< /Length {len(body)} >>\nstream\n{body}endstream".encode("latin-1"), + 5: b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>", + }) + + +def _mixed_pdf() -> bytes: + """Page 1 real text layer, page 2 chart image โ€” the production shape.""" + img = _chart_image() + buf = io.BytesIO() + img.save(buf, "JPEG", quality=88) + jpg, iw, ih = buf.getvalue(), img.width, img.height + + text = ( + "BT /F1 16 Tf 60 720 Td (Annual Report: Department of Ceramics) Tj ET\n" + "BT /F1 12 Tf 60 690 Td (The department operates a shared kiln facility rated for) Tj ET\n" + "BT /F1 12 Tf 60 672 Td (cone 10 reduction firing. The replacement cost of the) Tj ET\n" + "BT /F1 12 Tf 60 654 Td (primary kiln is estimated at 47,500 dollars as of this year.) Tj ET\n" + "BT /F1 12 Tf 60 624 Td (Quarterly enrollment is shown in the figure on page 2.) Tj ET\n" + ) + disp_w = 512.0 + disp_h = disp_w * ih / iw + stream = f"q {disp_w:.2f} 0 0 {disp_h:.2f} 50 400 cm /Im0 Do Q\n" + + return _assemble_pdf({ + 1: b"<< /Type /Catalog /Pages 2 0 R >>", + 2: b"<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 >>", + 3: b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] " + b"/Resources << /Font << /F1 7 0 R >> >> /Contents 5 0 R >>", + 4: b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] " + b"/Resources << /XObject << /Im0 8 0 R >> >> /Contents 6 0 R >>", + 5: f"<< /Length {len(text)} >>\nstream\n{text}endstream".encode("latin-1"), + 6: f"<< /Length {len(stream)} >>\nstream\n{stream}endstream".encode("latin-1"), + 7: b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>", + 8: (f"<< /Type /XObject /Subtype /Image /Width {iw} /Height {ih} " + f"/ColorSpace /DeviceRGB /BitsPerComponent 8 /Filter /DCTDecode " + f"/Length {len(jpg)} >>\nstream\n").encode("latin-1") + jpg + b"\nendstream", + }) + + +def build_corpus(out_dir: str) -> Dict[str, Tuple[str, str]]: + """Write the corpus, returning {key: (path, bedrock document name)}.""" + def image_pdf(img: Image.Image, name: str) -> str: + path = os.path.join(out_dir, name) + img.convert("RGB").save(path, "PDF", resolution=150.0) + return path + + def raw_pdf(data: bytes, name: str) -> str: + path = os.path.join(out_dir, name) + with open(path, "wb") as fh: + fh.write(data) + return path + + corpus = { + "chart": (image_pdf(_chart_image(), "chart_only.pdf"), "chart only"), + "table": (image_pdf(_table_image(), "table_in_image.pdf"), "table in image"), + "scan": (image_pdf(_scan_image(), "scanned_page.pdf"), "scanned page"), + "canary": (raw_pdf(_text_layer_pdf(), "text_layer.pdf"), "text layer canary"), + "mixed": (raw_pdf(_mixed_pdf(), "mixed_text_and_chart.pdf"), "annual report"), + } + for key, (path, _) in corpus.items(): + logger.info("corpus %-7s %-26s %8d bytes", key, os.path.basename(path), + os.path.getsize(path)) + return corpus + + +# ------------------------------------------------------------------ questions +# accept: any listed substring present -> correct +# accept_all: every listed substring must be present +# num: any integer in the answer inside (lo, hi) -> correct +# (chart bars carry no printed labels, so values are read off an +# axis and an exact-match bar would be unfair) +QUESTIONS: List[Dict[str, Any]] = [ + dict(id="c1", doc="chart", family="chart-value", num=(600, 660), truth="631", + q="What is the value of the Fall 2024 bar in this chart? Answer with the number only."), + dict(id="c2", doc="chart", family="chart-compare", accept=["spring 2024"], truth="Spring 2024", + q="Which period in this chart has the lowest enrollment? Answer with the period label only."), + dict(id="c3", doc="chart", family="chart-value", num=(300, 430), truth="~363", + q="Approximately how much higher is the Fall 2024 bar than the Spring 2024 bar? " + "Answer with a number only."), + dict(id="c4", doc="chart", family="chart-structure", + accept_all=["fall 2023", "spring 2024", "fall 2024", "spring 2025"], + truth="all four labels", + q="List the four period labels along the horizontal axis, left to right."), + dict(id="t1", doc="table", family="table-cell", accept=["401"], truth="$401", + q="What is the Materials fee for Metalsmithing? Answer with the amount only."), + dict(id="t2", doc="table", family="table-compare", accept=["printmaking"], truth="Printmaking", + q="Which program has the lowest Total? Answer with the program name only."), + dict(id="t3", doc="table", family="table-cell", accept=["118"], truth="$118", + q="What is the Lab Fee for Photography? Answer with the amount only."), + dict(id="s1", doc="scan", family="scan-fact", accept=["eleven", "11%", "11 percent"], + truth="eleven percent", + q="What percentage of each department's annual operating allocation is the " + "equipment replacement reserve held at?"), + dict(id="s2", doc="scan", family="scan-fact", accept=["pr-2291", "pr 2291", "pr2291"], + truth="PR-2291", + q="What is the document reference identifier shown on this memo?"), + dict(id="k1", doc="canary", family="text-layer-canary", accept=["3,400", "3400"], + truth="3,400 lb", + q="What is the rated capacity of the freight elevator? Answer with the number only."), + dict(id="m1", doc="mixed", family="mixed-text", accept=["47,500", "47500"], truth="$47,500 (p1)", + q="What is the estimated replacement cost of the primary kiln? Answer with the amount only."), + dict(id="m2", doc="mixed", family="mixed-chart", num=(600, 660), truth="631 (p2 image)", + q="In the figure, what is the value of the Fall 2024 bar? Answer with the number only."), + dict(id="m3", doc="mixed", family="mixed-chart", accept=["spring 2024"], + truth="Spring 2024 (p2)", + q="In the figure, which period has the lowest enrollment? Answer with the label only."), + dict(id="m4", doc="mixed", family="mixed-cross", accept_all=["yes"], truth="yes / yes", + q="Does the report state a kiln replacement cost, and does the figure show Fall 2024 " + "above 600? Answer yes/no to each."), +] + + +# -------------------------------------------------------------------- runner + +def document_block(corpus, doc_key: str, citations: bool) -> Dict[str, Any]: + """Mirror `DocumentHandler.create_content_block`, optionally + citations.""" + path, label = corpus[doc_key] + with open(path, "rb") as fh: + data = fh.read() + block: Dict[str, Any] = { + "document": {"format": "pdf", "name": label, "source": {"bytes": data}} + } + if citations: + block["document"]["citations"] = {"enabled": True} + return block + + +def extract(blocks: List[Dict[str, Any]]) -> Tuple[str, bool]: + """Pull answer text out of a response. + + With citations enabled the answer moves INSIDE ``citationsContent`` and the + top-level ``text`` blocks go empty โ€” a consumer reading only ``text`` sees + nothing. Returns (text, response carried citation blocks). + """ + parts: List[str] = [] + cited = False + for block in blocks: + if "text" in block: + parts.append(block["text"]) + if "citationsContent" in block: + cited = True + parts += [c.get("text", "") for c in block["citationsContent"].get("content", [])] + return " ".join(parts).strip(), cited + + +def score(question: Dict[str, Any], answer: str) -> bool: + lowered = answer.lower() + if "num" in question: + lo, hi = question["num"] + found = [int(n.replace(",", "")) for n in re.findall(r"\d[\d,]*", lowered)] + return any(lo <= n <= hi for n in found) + if "accept_all" in question: + return all(token in lowered for token in question["accept_all"]) + return any(token in lowered for token in question["accept"]) + + +def ask(client, model_id: str, corpus, question, citations: bool) -> Dict[str, Any]: + message = { + "role": "user", + "content": [document_block(corpus, question["doc"], citations), {"text": question["q"]}], + } + started = time.time() + error: Optional[str] = None + # Claude 5-family models reject `temperature` outright; fall back rather + # than scoring a 400 as a wrong answer. + for config in ({"maxTokens": 500, "temperature": 0}, {"maxTokens": 500}): + try: + response = client.converse(modelId=model_id, messages=[message], + inferenceConfig=config) + break + except Exception as exc: # noqa: BLE001 - probe reports, never raises + error = f"{type(exc).__name__}: {exc}" + if "temperature" not in str(exc): + return dict(error=error, answer="", cited=False) + else: + return dict(error=error, answer="", cited=False) + + text, cited = extract(response["output"]["message"]["content"]) + return dict( + error=None, + answer=text, + cited=cited, + refused=any(m in text.lower() for m in REFUSAL_MARKERS), + usage=response.get("usage", {}), + ms=int((time.time() - started) * 1000), + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("models", nargs="*", default=None, + help=f"Bedrock model ids (default: {DEFAULT_MODELS[0]})") + parser.add_argument("--out", default=None, help="directory for corpus + results JSON") + args = parser.parse_args() + + models = args.models or DEFAULT_MODELS + out_dir = args.out or tempfile.mkdtemp(prefix="g3-probe-") + os.makedirs(out_dir, exist_ok=True) + logger.info("corpus + results -> %s", out_dir) + + corpus = build_corpus(out_dir) + client = boto3.client("bedrock-runtime", region_name=REGION) + results: List[Dict[str, Any]] = [] + + for model_id in models: + print(f"\n{'=' * 86}\nMODEL {model_id}\n{'=' * 86}") + for question in QUESTIONS: + row: Dict[str, Any] = dict(model=model_id, id=question["id"], + doc=question["doc"], family=question["family"], + truth=question["truth"]) + for arm, citations in (("bare", False), ("cited", True)): + res = ask(client, model_id, corpus, question, citations) + res["correct"] = (not res["error"]) and score(question, res["answer"]) + row[arm] = res + bare, cited = row["bare"], row["cited"] + marker = "" if bare["correct"] == cited["correct"] else " <-- ARMS DIVERGE" + print(f"[{question['id']:>3}] {question['family']:<20} " + f"truth={question['truth']:<18} " + f"bare={'PASS' if bare['correct'] else 'fail'} " + f"cited={'PASS' if cited['correct'] else 'fail'}" + f"{'+cit' if cited['cited'] else ' '}{marker}") + if bare["error"] or cited["error"]: + print(f" ERROR bare={bare['error']} cited={cited['error']}") + results.append(row) + + results_path = os.path.join(out_dir, "results.json") + with open(results_path, "w") as fh: + json.dump(results, fh, indent=2, default=str) + + print(f"\n{'=' * 86}\nSUMMARY\n{'=' * 86}") + for model_id in models: + rows = [r for r in results if r["model"] == model_id] + bare_ok = sum(r["bare"]["correct"] for r in rows) + cited_ok = sum(r["cited"]["correct"] for r in rows) + with_cit = sorted(r["id"] for r in rows if r["cited"]["cited"]) + print(f"{model_id}") + print(f" bare {bare_ok}/{len(rows)} correct") + print(f" cited {cited_ok}/{len(rows)} correct") + print(f" responses carrying citation blocks: {len(with_cit)} {with_cit}") + print(f"\nresults -> {results_path}") + + +if __name__ == "__main__": + main() diff --git a/backend/src/agents/main_agent/chat_agent.py b/backend/src/agents/main_agent/chat_agent.py index 9d5b72f5f..91ced9959 100644 --- a/backend/src/agents/main_agent/chat_agent.py +++ b/backend/src/agents/main_agent/chat_agent.py @@ -84,6 +84,7 @@ async def stream_async( message: str, session_id: Optional[str] = None, files: Optional[List] = None, + attachment_names: Optional[List[str]] = None, citations: Optional[List] = None, original_message: Optional[str] = None, interrupt_responses: Optional[List[Dict[str, Any]]] = None, @@ -98,7 +99,12 @@ async def stream_async( `interrupt_responses` โ€” the paused turn already has the original prompt in `_interrupt_state`. session_id: Session identifier (defaults to instance session_id) - files: Optional list of FileContent objects (with base64 bytes) + files: Optional list of FileContent objects (with base64 bytes). + Inline attachments only โ€” diverted ones (spreadsheets, decks) + must not be here or they become invalid document blocks. + attachment_names: Every filename the user attached this turn, + including diverted ones, for the `[Attached files: โ€ฆ]` marker + the SPA replays to rebuild attachment cards on reload. citations: Optional list of citation dicts from RAG retrieval original_message: Original user message before RAG augmentation interrupt_responses: When set, resume a paused agent turn by @@ -129,7 +135,9 @@ async def stream_async( # user turn, no multimodal/files. prompt = [] else: - prompt = self.multimodal_builder.build_prompt(message, files) + prompt = self.multimodal_builder.build_prompt( + message, files, attachment_names=attachment_names + ) async for event in self.stream_coordinator.stream_response( agent=self.agent, diff --git a/backend/src/agents/main_agent/integrations/external_mcp_client.py b/backend/src/agents/main_agent/integrations/external_mcp_client.py index 5b3f485e0..051c34629 100644 --- a/backend/src/agents/main_agent/integrations/external_mcp_client.py +++ b/backend/src/agents/main_agent/integrations/external_mcp_client.py @@ -15,7 +15,7 @@ import logging import re -from typing import Any, Callable, Optional, List, Set +from typing import Any, Callable, Iterator, Optional, List, Set from urllib.parse import urlparse from mcp.client.streamable_http import streamablehttp_client @@ -126,6 +126,64 @@ def detect_aws_service_from_url(url: str) -> Optional[str]: return None +# HTTP statuses that mean "this caller is not authorized" โ€” the only kind of +# pre-flight failure that completing an OAuth consent flow can actually fix. +_AUTH_STATUS_CODES = frozenset({401, 403}) + +# Fallback for transports that stringify the status instead of raising an +# httpx error carrying a `.response` (an MCP server may surface the refusal as +# a protocol-level message). Deliberately narrow: matching bare digits would +# fire on any URL or port that happens to contain them. +_AUTH_TEXT_RE = re.compile( + r"\b(?:401|403)\b|\bunauthorized\b|\bforbidden\b", re.IGNORECASE +) + + +def _iter_exception_chain(exc: BaseException) -> Iterator[BaseException]: + """Yield `exc` and everything reachable through its cause/context chain + and any ExceptionGroup members, each exactly once. + + The exception we catch is never the interesting one: Strands wraps the + real failure twice (`ToolProviderException` around + `MCPClientInitializationError`), and the transport or HTTP error + underneath arrives inside an anyio `ExceptionGroup`. + """ + seen: set[int] = set() + stack: list[BaseException] = [exc] + while stack: + e = stack.pop() + if id(e) in seen: + continue + seen.add(id(e)) + yield e + for nxt in (e.__cause__, e.__context__): + if nxt is not None: + stack.append(nxt) + stack.extend(getattr(e, "exceptions", ()) or ()) # ExceptionGroup + + +def _is_auth_failure(exc: BaseException) -> bool: + """True when `exc`'s chain carries an HTTP 401/403. + + Used to decide whether a failed pre-flight is plausibly a *consent* gap. + A connection refusal, DNS failure, or timeout is not: the server simply + isn't answering, and no amount of authorizing will change that. Treating + those as consent gaps prompts the user to connect a server that isn't + there, on every single turn, with no way for the prompt to succeed. + """ + for e in _iter_exception_chain(exc): + response = getattr(e, "response", None) + for code in ( + getattr(response, "status_code", None), + getattr(e, "status_code", None), + ): + if isinstance(code, int) and code in _AUTH_STATUS_CODES: + return True + if _AUTH_TEXT_RE.search(str(e)): + return True + return False + + def create_external_mcp_client( config: MCPServerConfig, tool_definition: Optional[ToolDefinition] = None, @@ -306,6 +364,133 @@ def __init__(self): # than a global tool-name list (so two MCP servers can share a tool # name and only one is gated). self._approval_names_for_client_id: dict[int, set[str]] = {} + # user_id -> {provider_id: authorization_url} for OAuth-gated tools + # whose pre-flight failed because the user has not consented yet. + # Drained by the stream coordinator (`take_pending_consents`) so the + # turn can surface an `oauth_required` event instead of dropping the + # tool silently. Keyed by user because this integration is a + # process-wide singleton shared across concurrent sessions. + self._pending_consents: dict[str, dict[str, str]] = {} + + def take_pending_consents(self, user_id: str) -> dict[str, str]: + """Pop and return {provider_id: authorization_url} for `user_id`. + + Draining on read keeps the singleton from re-emitting a stale prompt + on a later turn: if the user still hasn't consented, the next + `load_external_tools` fails pre-flight again and re-records it. On an + agent-cache hit no loading happens, nothing is recorded, and nothing + is emitted โ€” correct, because the prompt already went out once. + """ + return self._pending_consents.pop(user_id, {}) + + async def _recover_oauth_preflight( + self, + *, + tool_id: str, + client: MCPClient, + user_id: Optional[str], + provider_id: Optional[str], + exc: Exception, + ) -> bool: + """Second chance for an OAuth-gated tool whose pre-flight failed. + + A server that requires auth even for `tools/list` (GitHub's MCP + endpoint does) 401s whenever the in-process token cache is cold โ€” + which it always is on a fresh microVM. Before this recovery step the + tool was dropped, and because `OAuthConsentHook` only runs + `BeforeToolCall` for *registered* tools, nothing ever warmed the + cache: the tool stayed missing for the life of the process even for + a user whose token was sitting in the AgentCore vault the whole time. + + Only an *auth* failure qualifies. A pre-flight that failed because the + server is unreachable, timed out, or resolved to nothing is not a + consent gap โ€” asking the vault there would answer "no token, here's an + authorization URL" for a server that isn't listening, and the user + would be told to connect it on every turn with no way to succeed. + + So on an auth failure we ask the vault directly: + * token -> warm the cache and retry the pre-flight once. This is + the consented-user path, and it is why the tool comes back by + itself after the user completes consent in the popup. + * consent URL -> AgentCore is telling us this user genuinely has + not authorized. Record it so the turn can emit `oauth_required` + rather than dropping the tool with no explanation. + + Returns True when the client is usable and should be registered. + """ + if not provider_id or not user_id: + logger.warning( + f"Skipping external MCP tool {tool_id}: " + f"failed to start client ({exc})" + ) + return False + + # Only meaningful when the cache was cold. A warm token that still + # failed is an expiry/refresh case, which `OAuthConsentHook`'s + # AfterToolCall 401 handler already owns โ€” re-asking the vault with + # force_authentication=False would just hand back the same token. + if oauth_token_cache.get(user_id, provider_id): + logger.warning( + f"Skipping external MCP tool {tool_id}: failed to start client " + f"with a cached {provider_id} token ({exc})" + ) + return False + + # Consent can only fix a refusal to authorize. Anything else โ€” the + # server is down, the URL is wrong, the host doesn't resolve โ€” keeps + # the pre-recovery behaviour of dropping the tool quietly. + if not _is_auth_failure(exc): + logger.warning( + f"Skipping external MCP tool {tool_id}: pre-flight failed without " + f"an auth error, so this is not a {provider_id} consent gap " + f"({exc})" + ) + return False + + from apis.shared.oauth.token_resolution import resolve_token_or_consent_url + + resolved = await resolve_token_or_consent_url(provider_id, user_id) + if resolved is None: + # Couldn't ask AgentCore at all โ€” not evidence of a consent gap, + # so stay silent rather than prompting for a connector that may + # well be authorized. + logger.warning( + f"Skipping external MCP tool {tool_id}: failed to start client " + f"and could not resolve a {provider_id} token ({exc})" + ) + return False + + if resolved["token"]: + oauth_token_cache.set(user_id, provider_id, resolved["token"]) + try: + await client.load_tools() + except Exception as retry_exc: + logger.warning( + f"Skipping external MCP tool {tool_id}: pre-flight still failed " + f"after warming the {provider_id} token from the vault " + f"({retry_exc})" + ) + return False + logger.info( + f"Recovered external MCP tool {tool_id}: warmed {provider_id} " + "token from the AgentCore vault after a cold-cache pre-flight failure" + ) + return True + + authorization_url = resolved["url"] + if not authorization_url: + logger.warning( + f"Skipping external MCP tool {tool_id}: no {provider_id} token and " + f"no authorization URL from AgentCore ({exc})" + ) + return False + + self._pending_consents.setdefault(user_id, {})[provider_id] = authorization_url + logger.info( + f"External MCP tool {tool_id} needs {provider_id} consent; " + "surfacing oauth_required instead of dropping it silently" + ) + return False def _get_cache_key(self, tool_id: str, user_id: Optional[str], requires_oauth: bool) -> str: if requires_oauth and user_id: @@ -497,11 +682,15 @@ async def _exchange( try: await client.load_tools() except Exception as exc: - logger.warning( - f"Skipping external MCP tool {tool_id}: " - f"failed to start client ({exc})" + recovered = await self._recover_oauth_preflight( + tool_id=tool_id, + client=client, + user_id=user_id, + provider_id=provider_id, + exc=exc, ) - continue + if not recovered: + continue self.clients[cache_key] = client self._client_versions[cache_key] = tool_version diff --git a/backend/src/agents/main_agent/multimodal/prompt_builder.py b/backend/src/agents/main_agent/multimodal/prompt_builder.py index 2b254aaa6..23660915c 100644 --- a/backend/src/agents/main_agent/multimodal/prompt_builder.py +++ b/backend/src/agents/main_agent/multimodal/prompt_builder.py @@ -23,33 +23,52 @@ def __init__(self): def build_prompt( self, message: str, - files: Optional[List[Any]] = None + files: Optional[List[Any]] = None, + attachment_names: Optional[List[str]] = None, ) -> Union[str, List[Dict[str, Any]]]: """ Build prompt for Strands Agent with multimodal support Args: message: User message text - files: Optional list of FileContent objects with base64 bytes + files: Optional list of FileContent objects with base64 bytes. + These become content blocks, so this must be the *inline* + set only โ€” handing it a .pptx would produce a document block + in a format Bedrock's enum doesn't accept. + attachment_names: Optional authoritative filename list for the + ``[Attached files: โ€ฆ]`` marker. Defaults to the names in + ``files``. Pass this when some attachments are deliberately + kept out of ``files`` (spreadsheets route to the analysis + tools, decks to the PowerPoint tools) โ€” the marker is what + the SPA replays to rebuild attachment cards on reload, so a + name missing here means that file's card silently disappears + from history even though the file itself is still in the + session. Returns: str or list[ContentBlock]: Simple string or multimodal content blocks """ - # If no files, return simple text + marker_names = ( + list(attachment_names) + if attachment_names is not None + else [f.filename for f in (files or []) if hasattr(f, 'filename')] + ) + + # The marker must stay at the very END of the text: the SPA's + # ATTACHED_FILES_PATTERN is `$`-anchored. + text = message + if marker_names: + text = f"{message}\n\n[Attached files: {', '.join(marker_names)}]" + + # Nothing to inline โ€” return plain text. This still carries the + # marker, which is the case where every attachment was diverted + # (e.g. a lone .pptx): previously this returned early with a bare + # message and the attachment vanished from restored history. if not files or len(files) == 0: - return message + return text # Build ContentBlock list for multimodal input - content_blocks = [] - - # Add text first (with file reference marker for session history reconstruction) - file_names = [f.filename for f in files if hasattr(f, 'filename')] - if file_names: - # Add file reference marker after user message for session history - text_with_marker = f"{message}\n\n[Attached files: {', '.join(file_names)}]" - content_blocks.append({"text": text_with_marker}) - else: - content_blocks.append({"text": message}) + content_blocks: List[Dict[str, Any]] = [{"text": text}] # Track sanitized document names used in this turn to prevent # Bedrock ValidationException: "Messages can't contain duplicate document names" diff --git a/backend/src/agents/main_agent/streaming/stream_coordinator.py b/backend/src/agents/main_agent/streaming/stream_coordinator.py index ae6916ab4..3bb4d04ef 100644 --- a/backend/src/agents/main_agent/streaming/stream_coordinator.py +++ b/backend/src/agents/main_agent/streaming/stream_coordinator.py @@ -69,6 +69,135 @@ def reset_cancellation_state(agent: Any, session_manager: Any) -> None: cancel_signal.clear() +def _is_interrupt_resume_prompt(prompt: Any) -> bool: + """True when `prompt` is Strands' resume payload for a paused turn. + + Mirrors ``strands.interrupt.InterruptState.resume``'s own acceptance test + โ€” a list whose every content block carries nothing but ``interruptResponse`` + โ€” so we can never disagree with it about what counts as a resume. + + One deliberate difference: an EMPTY list is not a resume here. Strands + tolerates it (``all()`` over nothing is True), but in this codebase ``[]`` + is the max_tokens "Continue" prompt (``chat_agent.py``), and ``if + interrupt_responses:`` means a real resume always carries at least one + entry. Treating ``[]`` as a resume would leave a stale pause armed on a + continuation. + """ + if not isinstance(prompt, list) or not prompt: + return False + return all( + isinstance(content, dict) + and content + and all(key == "interruptResponse" for key in content) + for content in prompt + ) + + +def _message_has_tool_use(message: Any) -> bool: + if not isinstance(message, dict): + return False + return any( + isinstance(block, dict) and "toolUse" in block + for block in (message.get("content") or []) + ) + + +def _drop_abandoned_turn_tail(messages: List[Dict[str, Any]]) -> int: + """Pop trailing messages until history ends on a completed assistant turn. + + Mutates in place and returns the number dropped. In-place is mandatory: + the message list is **aliased** across the cached agents serving one + session (#741/#750, see ``_adopt_session_conversation``), and rebinding + mid-life silently breaks that alias. + """ + dropped = 0 + while messages: + last = messages[-1] + if ( + isinstance(last, dict) + and last.get("role") == "assistant" + and not _message_has_tool_use(last) + ): + break + messages.pop() + dropped += 1 + return dropped + + +def reset_stale_interrupt_state(agent: Any, prompt: Any) -> None: + """Abandon a pause left by a PREVIOUS turn when this turn isn't a resume. + + When ``OAuthConsentHook`` (or the tool-approval hook) calls + ``event.interrupt(...)``, Strands sets ``_interrupt_state.activated`` and + stops. If the user never completes consent and instead just types a new + message, that flag is still set on the cached agent โ€” and + ``InterruptState.resume`` rejects a plain string prompt with + ``TypeError: prompt_type= | must resume from interrupt with + list of interruptResponse's``. It reached the user as a non-recoverable + ``stream_error``: the session was stuck, because every subsequent fresh + turn hit the same flag. + + The "a fresh turn supersedes a paused turn" policy already exists โ€” see + ``clear_paused_turn`` / ``clear_interrupted_turn`` in + ``inference_api/chat/routes.py``. Those only clear the DynamoDB side; the + live object on the cached agent was missed. Same sticky-state-on-a-cached- + agent family as the cancel flags above. + + Deactivating is not enough on its own. Strands appends the assistant + ``toolUse`` message to ``agent.messages`` *before* running tools + (``event_loop.py``), and on interrupt it returns without ever appending + the matching ``toolResult`` โ€” so history ends on an unanswered tool call. + ``_repair_tool_pairing`` can't help here: it deliberately leaves a + *trailing* toolUse alone ("left for prompt-arrival handling") and doesn't + even count it as a violation, so at this point in the turn it no-ops. Left + as-is, Strands appends the new user message behind the dangling toolUse + and Bedrock rejects the request. + + So we drop the abandoned turn back to the last completed assistant turn. + That clears the unanswered toolUse *and* leaves the history ending on an + assistant message, so the incoming user prompt keeps roles alternating. + Synthesizing an error ``toolResult`` instead would satisfy the pairing + rule but leave two consecutive user turns (the synthetic result, then the + real prompt), which Bedrock rejects just the same. + + Dropping messages rewrites the prompt-cache prefix, so this costs a cache + write โ€” on a turn the user has already abandoned, which is the right place + to spend it. Nothing is lost from the *conversation*: the abandoned turn + produced no assistant answer, and the transcript the user sees is + persisted separately. + """ + interrupt_state = getattr(agent, "_interrupt_state", None) + if interrupt_state is None or not getattr(interrupt_state, "activated", False): + return + + if _is_interrupt_resume_prompt(prompt): + return + + logger.info( + "Abandoning a paused turn: this turn is not an interrupt resume " + "(%d pending interrupt(s))", + len(getattr(interrupt_state, "interrupts", None) or {}), + ) + + try: + interrupt_state.deactivate() + except Exception: + logger.exception("Failed to deactivate stale interrupt state") + return + + messages = getattr(agent, "messages", None) + if not isinstance(messages, list) or not messages: + return + + dropped = _drop_abandoned_turn_tail(messages) + if dropped: + logger.info( + "Dropped %d message(s) from the abandoned turn so the incoming " + "prompt lands on a valid history", + dropped, + ) + + class StreamCoordinator: """Coordinates streaming lifecycle for agent responses""" @@ -130,6 +259,12 @@ async def stream_response( # brick this one. See ``reset_cancellation_state``. reset_cancellation_state(agent, session_manager) + # Likewise a pause armed by a previous turn: if the user abandoned an + # OAuth/tool-approval consent and just typed again, the still-armed + # interrupt state makes Strands reject this turn's prompt outright. + # See ``reset_stale_interrupt_state``. + reset_stale_interrupt_state(agent, prompt) + # Track timing for latency metrics stream_start_time = time.time() # Wall-clock turn start as a tz-aware datetime. Used post-turn to @@ -440,6 +575,8 @@ async def stream_response( user_id=user_id, ): yield sse + for sse in self._extract_preflight_consent_events(user_id): + yield sse # Check if this is the "done" event - send final metadata before it if event.get("type") == "done": @@ -1426,6 +1563,53 @@ async def _persist_paused_turn_snapshot( session_id, e, exc_info=True, ) + def _extract_preflight_consent_events( + self, + user_id: Optional[str] = None, + ) -> List[str]: + """Yield an `oauth_required` event per OAuth-gated MCP tool that was + dropped at agent-build time because the user hasn't consented. + + These are NOT Strands interrupts โ€” the turn ran to completion, just + without the tool, because an unauthorized `tools/list` meant we never + learned what the server exposes and so could never advertise it to + the model. The events therefore carry no `interruptId` and the + frontend must not try to resume anything; it shows the Connect + affordance, and the tool registers by itself on the next turn once + `_recover_oauth_preflight` can warm a real token from the vault. + + Deliberately not persisted as a `pending_interrupt` breadcrumb: those + are keyed by interrupt id for the resume path, and a refresh doesn't + need one here โ€” while consent is still outstanding, the next turn + fails pre-flight again and re-emits this event. + """ + if not user_id: + return [] + + from agents.main_agent.integrations.external_mcp_client import ( + get_external_mcp_integration, + ) + from apis.shared.oauth.models import OAuthRequiredEvent + + try: + pending = get_external_mcp_integration().take_pending_consents(user_id) + except Exception: + logger.exception("Failed to read pre-flight OAuth consents") + return [] + + events: List[str] = [] + for provider_id, authorization_url in sorted(pending.items()): + logger.info( + "Emitting pre-flight oauth_required for provider=%s", provider_id + ) + events.append( + OAuthRequiredEvent( + provider_id=provider_id, + authorization_url=authorization_url, + ).to_sse_format() + ) + return events + async def _extract_oauth_required_events( self, agent: Any, diff --git a/backend/src/apis/app_api/files/routes.py b/backend/src/apis/app_api/files/routes.py index 4ec912007..9985ee865 100644 --- a/backend/src/apis/app_api/files/routes.py +++ b/backend/src/apis/app_api/files/routes.py @@ -62,10 +62,11 @@ async def request_presigned_url( 2. Use the returned presignedUrl to PUT the file directly to S3 3. Call POST /files/{uploadId}/complete to finalize - **Supported file types:** PDF, DOCX, TXT, HTML, CSV, XLS, XLSX, MD + **Supported file types:** PDF, DOCX, TXT, HTML, CSV, XLS, XLSX, PPTX, MD **Limits:** - - Maximum file size: 4MB + - Maximum file size: 4MB (25MB for PPTX โ€” decks never go inline to + Bedrock, so the inline-document ceiling doesn't bound them) - Maximum files per message: 5 - User storage quota: 1GB """ diff --git a/backend/src/apis/app_api/files/service.py b/backend/src/apis/app_api/files/service.py index b9b136ce2..39474b745 100644 --- a/backend/src/apis/app_api/files/service.py +++ b/backend/src/apis/app_api/files/service.py @@ -32,6 +32,7 @@ FileListResponse, QuotaResponse, is_allowed_mime_type, + is_presentation_file, ALLOWED_MIME_TYPES, ) from .thumbnails import ( @@ -125,6 +126,7 @@ def __init__( s3_client=None, bucket_name: Optional[str] = None, max_file_size: Optional[int] = None, + presentation_max_file_size: Optional[int] = None, max_files_per_message: Optional[int] = None, user_quota_bytes: Optional[int] = None, thumbnail_renderer: Optional[ThumbnailRenderer] = None, @@ -156,6 +158,22 @@ def __init__( self.max_file_size = max_file_size or int( os.environ.get("FILE_UPLOAD_MAX_SIZE_BYTES", 4 * 1024 * 1024) # 4MB ) + # Presentations get a larger cap than the general limit. The general + # 4MB is sized for what Bedrock will accept as an inline document + # block; a .pptx never goes inline (Bedrock has no pptx document + # format โ€” see `is_presentation_file`), so that ceiling does not + # apply to it. What does apply is the Code Interpreter hop in the + # PowerPoint tools: `_ci_write_bytes` base64-encodes the whole deck + # into a single writeFiles `text` field, inflating it ~4/3. That + # field is a MaxLenString (100MB), so 25MB of deck โ†’ ~33MB of base64 + # stays well inside it. Real-world corporate templates with imagery + # routinely exceed 4MB, which is what made the template workflow + # unusable at the general cap. + self.presentation_max_file_size = presentation_max_file_size or int( + os.environ.get( + "FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION", 25 * 1024 * 1024 # 25MB + ) + ) self.max_files_per_message = max_files_per_message or int( os.environ.get("FILE_UPLOAD_MAX_FILES_PER_MESSAGE", 5) ) @@ -167,6 +185,18 @@ def __init__( self.presign_expiration = 15 * 60 # 15 minutes self.preview_url_expiration = 10 * 60 # 10 minutes for GET previews + def max_size_for(self, filename: str, mime_type: str) -> int: + """Return the upload size cap that applies to this file. + + Presentations get their own (larger) cap; everything else gets the + general limit. Callers must use this rather than reading + `max_file_size` directly, or a .pptx under the presentation cap gets + rejected by whichever gate was missed. + """ + if is_presentation_file(filename, mime_type): + return self.presentation_max_file_size + return self.max_file_size + # ========================================================================= # Pre-signed URL Flow # ========================================================================= @@ -200,9 +230,10 @@ async def request_presigned_url( if not is_allowed_mime_type(request.mime_type): raise InvalidFileTypeError(request.mime_type) - # Validate file size - if request.size_bytes > self.max_file_size: - raise FileTooLargeError(request.size_bytes, self.max_file_size) + # Validate file size against the cap for this file's class + size_limit = self.max_size_for(request.filename, request.mime_type) + if request.size_bytes > size_limit: + raise FileTooLargeError(request.size_bytes, size_limit) # Check quota quota = await self.repository.get_user_quota(user_id) diff --git a/backend/src/apis/app_api/sessions/routes.py b/backend/src/apis/app_api/sessions/routes.py index c161f191a..9ced1f9d2 100644 --- a/backend/src/apis/app_api/sessions/routes.py +++ b/backend/src/apis/app_api/sessions/routes.py @@ -647,22 +647,32 @@ async def signal_turn_interrupted_endpoint( body: SessionInterruptRequest, current_user: User = Depends(get_current_user_from_session), ): - """Record that the user deliberately stopped the session's in-flight turn. + """Record a client-attested reason for the session's turn being interrupted. - This is the AUTHORITATIVE carrier of stop intent for the interrupted-turn - flow: the transport cannot distinguish a Stop click from a dropped socket - (both surface as a cancelled stream), so the SPA signals intent here - out-of-band when the user clicks Stop โ€” via ``fetch(..., {keepalive: + This is the AUTHORITATIVE carrier of client intent for the + interrupted-turn flow: the transport cannot distinguish a Stop click from + a refresh from a dropped socket (all three surface as a cancelled + stream), so the SPA signals out-of-band โ€” via ``fetch(..., {keepalive: true})`` with the ``X-CSRF-Token`` header (NOT ``navigator.sendBeacon``, which cannot set headers and would be rejected by CSRFMiddleware). + Two reasons are accepted, and they mean different things: + + * ``user_stopped`` โ€” the Stop button. Deliberate: the user rejected + the response in flight, so the turn is cancelled server-side too. + * ``navigated_away`` โ€” the page was hidden or unloaded mid-turn. The + user left; they did not reject anything. **Recorded only** โ€” the + running turn is deliberately left alone, matching today's behaviour + where a refresh lets the turn finish server-side and the reload + offers to continue it. + Lives on app-api, not inference-api: the AgentCore Runtime data plane only proxies ``/invocations`` + ``/ping``, so a custom inference-api route would 404 in cloud. - ``user_stopped`` takes precedence over the ``connection_lost`` fallback - that inference-api's cancellation backstop may race against this write - (see ``set_interrupted_turn``). No-op for missing sessions โ€” and the GSI + Both take precedence over the ``connection_lost`` fallback that + inference-api's cancellation backstop may race against this write (see + ``set_interrupted_turn``). No-op for missing sessions โ€” and the GSI lookup inside ``set_interrupted_turn`` is user-scoped, so a session owned by someone else is also a no-op. Returns 204 either way (the user's intent is recorded best-effort; the client never waits on it). @@ -685,11 +695,19 @@ async def signal_turn_interrupted_endpoint( # so the user's resend isn't rejected with 409 and stopping wasted # model/tool work. Owner-scoped, so a stale Stop can't kill a later # turn. Best-effort: never fail the Stop signal on this. - try: - from apis.shared.sessions.session_lease import request_session_cancel - await request_session_cancel(session_id, user_id) - except Exception: - logger.warning("Failed to arm session cancel on stop", exc_info=True) + # + # Deliberate Stop ONLY. `navigated_away` is an attribution signal, not + # an instruction: cancelling on it would make every refresh kill the + # turn it interrupted, discarding work the reload is about to offer to + # continue. Leaving the turn running preserves exactly today's + # behaviour for a departure โ€” this endpoint's reason set widened, the + # side effects did not. + if body.reason == "user_stopped": + try: + from apis.shared.sessions.session_lease import request_session_cancel + await request_session_cancel(session_id, user_id) + except Exception: + logger.warning("Failed to arm session cancel on stop", exc_info=True) return Response(status_code=204) except Exception: logger.error("Error recording turn interruption", exc_info=True) diff --git a/backend/src/apis/inference_api/chat/routes.py b/backend/src/apis/inference_api/chat/routes.py index 75d35c888..91ff8d71a 100644 --- a/backend/src/apis/inference_api/chat/routes.py +++ b/backend/src/apis/inference_api/chat/routes.py @@ -714,27 +714,43 @@ def _estimate_decoded_size(file: "FileContent") -> int: def _partition_attachments( all_files: list, -) -> tuple[list, list, list]: - """Split attachments into (inline_for_bedrock, tabular, oversized_non_tabular). +) -> tuple[list, list, list, list]: + """Split attachments into + (inline_for_bedrock, tabular, presentations, oversized_non_tabular). - Tabular files (csv/xlsx) are never sent inline โ€” they route through the spreadsheet analysis tools. Keeps Bedrock's 4.5MB document limit from exploding on XLSX files that expand during internal parsing. + - Presentations (pptx) are never sent inline either, but for a harder + reason: Bedrock's document-block format enum has no `pptx`, so an + inline deck is a guaranteed ValidationException. They route through + the PowerPoint tools (see `is_presentation_file`). - Non-tabular files larger than INLINE_DOCUMENT_MAX_BYTES are dropped from the inline set with a user-facing note, to prevent mid-stream ValidationException on the raw AWS error path. - Everything else rides along as a regular document/image content block. + + Both diverted classes are checked before the size gate: they never go + inline at any size, so an oversized note would misdescribe them. """ - from apis.shared.files.models import INLINE_DOCUMENT_MAX_BYTES, is_tabular_file + from apis.shared.files.models import ( + INLINE_DOCUMENT_MAX_BYTES, + is_presentation_file, + is_tabular_file, + ) inline: list = [] tabular: list = [] + presentations: list = [] oversized: list = [] for file in all_files: if is_tabular_file(file.filename, file.content_type): tabular.append(file) continue + if is_presentation_file(file.filename, file.content_type): + presentations.append(file) + continue # Only size-gate non-image documents. Images have their own Bedrock # limits (much larger) and the prompt builder reroutes them as # image blocks, which are not affected by the document-size cap. @@ -745,11 +761,38 @@ def _partition_attachments( continue inline.append(file) - return inline, tabular, oversized + return inline, tabular, presentations, oversized + + +def _attachment_marker_names(all_files: list, oversized_inline: list) -> list: + """Filenames for the ``[Attached files: โ€ฆ]`` marker on the user message. + + The SPA replays that marker on session load to rebuild attachment cards + (``restoreFileAttachments``): the ``fileAttachment`` content block it + renders from is built client-side at send time and is never persisted, so + the marker is the only surviving link between a file and the message it + was attached to. + + Deliberately NOT ``files_to_send``. Diverted spreadsheets and decks are + still in the session and still reachable through their tools, so their + cards have to survive a reload too โ€” deriving this from the inline set + alone is exactly what made an uploaded .pptx vanish from history while + the file itself remained perfectly present. + + Oversized files are excluded: those were dropped from the turn entirely + and the guidance text already explains their absence. + + Order follows ``all_files`` (how the user attached them) rather than a + concatenation of the partition buckets, so the text is deterministic โ€” + it lands in the cacheable prefix on every later turn. + """ + oversized_names = {f.filename for f in oversized_inline} + return [f.filename for f in all_files if f.filename not in oversized_names] def _build_attachment_guidance( diverted_tabular: list, + diverted_presentations: list, oversized_inline: list, enabled_tools: list | None, ) -> str: @@ -779,6 +822,27 @@ def _build_attachment_guidance( f"to the message input), then re-send your message._" ) + if diverted_presentations: + names = ", ".join(f"`{f.filename}`" for f in diverted_presentations) + tool_is_enabled = bool(enabled_tools) and bool( + POWERPOINT_PRESENTATION_TOOL_IDS.intersection(enabled_tools) + ) + if tool_is_enabled: + parts.append( + f"_Attached presentation(s) {names} are available through the " + f"PowerPoint Presentations tool rather than inline โ€” use " + f"`read_powerpoint_presentation` to read a deck's slide text " + f"and speaker notes, or pass it as `template_name` to " + f"`create_powerpoint_presentation` to build on its layouts._" + ) + else: + parts.append( + f"_Attached presentation(s) {names} can't be read inline. To " + f"work with them, enable **PowerPoint Presentations** in the " + f"Tools section of the settings panel (gear icon next to the " + f"message input), then re-send your message._" + ) + if oversized_inline: names = ", ".join(f"`{f.filename}`" for f in oversized_inline) parts.append( @@ -1238,6 +1302,11 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # (1.4MB zipped โ†’ >4.5MB internal, triggering ValidationException). # They remain available to the agent via list_spreadsheets / # analyze_spreadsheet, which run pandas on the real file. See #206. + # - presentation_files: pptx, also never sent inline โ€” Bedrock's + # document-block format enum has no `pptx` member at all, so an + # inline deck is an unconditional ValidationException. They remain + # available via list/read_powerpoint_presentation, which extract + # slide text with python-pptx in Code Interpreter. # - oversized_files: non-tabular docs that exceed our inline size # budget; we skip them inline and surface a note instead of # letting Bedrock reject the turn. @@ -1289,18 +1358,31 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g ) all_files = deduped_files - files_to_send, diverted_tabular, oversized_inline = _partition_attachments(all_files) + ( + files_to_send, + diverted_tabular, + diverted_presentations, + oversized_inline, + ) = _partition_attachments(all_files) if diverted_tabular: logger.info( f"Diverted {len(diverted_tabular)} tabular file(s) from inline document blocks; " f"available via spreadsheet tools: {[f.filename for f in diverted_tabular]}" ) + if diverted_presentations: + logger.info( + f"Diverted {len(diverted_presentations)} presentation(s) from inline document " + f"blocks (Bedrock has no pptx document format); available via PowerPoint tools: " + f"{[f.filename for f in diverted_presentations]}" + ) if oversized_inline: logger.warning( f"Skipped {len(oversized_inline)} oversized file(s) (> inline limit): " f"{[(f.filename, _estimate_decoded_size(f)) for f in oversized_inline]}" ) + attachment_marker_names = _attachment_marker_names(all_files, oversized_inline) + # Pre-create session metadata so OAuth interrupts and other state can # attach to the session row from turn one. Best-effort; on failure the # post-stream lazy-create in StreamCoordinator still covers it. @@ -2217,7 +2299,10 @@ def _session_title_sse() -> Optional[str]: # The original text becomes the single source of truth for UI display, # while the full augmented prompt stays in AgentCore Memory for the LLM. attachment_guidance = _build_attachment_guidance( - diverted_tabular, oversized_inline, effective_enabled_tools + diverted_tabular, + diverted_presentations, + oversized_inline, + effective_enabled_tools, ) # When multiple spreadsheets are visible, ship the full inventory # up front so the agent can disambiguate intentionally instead of @@ -2266,6 +2351,10 @@ def _session_title_sse() -> Optional[str]: message_will_be_modified = ( final_message != input_data.message # RAG augmentation / attachment guidance / inventory or bool(files_to_send) # File attachments + # The `[Attached files: โ€ฆ]` marker is appended for diverted + # attachments too, so the persisted text differs from what the + # user typed even when nothing went inline (a lone .pptx). + or bool(attachment_marker_names) ) # Strands' resume protocol wants each entry wrapped as # {"interruptResponse": {...}}. The InvocationRequest schema @@ -2281,6 +2370,7 @@ def _session_title_sse() -> Optional[str]: final_message, session_id=input_data.session_id, files=files_to_send if files_to_send else None, + attachment_names=attachment_marker_names or None, citations=citations_for_storage if citations_for_storage else None, original_message=input_data.message if message_will_be_modified else None, interrupt_responses=interrupt_responses_payload, diff --git a/backend/src/apis/shared/files/__init__.py b/backend/src/apis/shared/files/__init__.py index de83b81e5..9a12684b1 100644 --- a/backend/src/apis/shared/files/__init__.py +++ b/backend/src/apis/shared/files/__init__.py @@ -19,10 +19,13 @@ ALLOWED_EXTENSIONS, TABULAR_MIME_TYPES, TABULAR_EXTENSIONS, + PRESENTATION_MIME_TYPES, + PRESENTATION_EXTENSIONS, INLINE_DOCUMENT_MAX_BYTES, get_file_format, is_allowed_mime_type, is_tabular_file, + is_presentation_file, ) from .repository import ( @@ -64,10 +67,13 @@ "ALLOWED_EXTENSIONS", "TABULAR_MIME_TYPES", "TABULAR_EXTENSIONS", + "PRESENTATION_MIME_TYPES", + "PRESENTATION_EXTENSIONS", "INLINE_DOCUMENT_MAX_BYTES", "get_file_format", "is_allowed_mime_type", "is_tabular_file", + "is_presentation_file", # Repository "FileUploadRepository", "get_file_upload_repository", diff --git a/backend/src/apis/shared/files/models.py b/backend/src/apis/shared/files/models.py index 2aa42a8e9..ae0b7459c 100644 --- a/backend/src/apis/shared/files/models.py +++ b/backend/src/apis/shared/files/models.py @@ -112,6 +112,49 @@ def is_tabular_file(filename: str, mime_type: str) -> bool: return False +# ============================================================================= +# Presentation File Detection +# ============================================================================= + +# PowerPoint files are routed to the PowerPoint presentation tools +# (list_powerpoint_presentations, read_powerpoint_presentation) instead of +# being sent inline as Bedrock document content blocks. Unlike the tabular +# carve-out below, this one is not a size optimization โ€” it is mandatory: +# Bedrock's Converse `DocumentFormat` enum has no `pptx` member (pdf, csv, +# doc, docx, xls, xlsx, html, txt, md are the only accepted values), so an +# inline .pptx fails the whole turn with a ValidationException. The tools are +# the only path that works. +# +# read_powerpoint_presentation extracts slide text, tables and speaker notes +# with python-pptx inside Code Interpreter, which is also far cheaper in +# tokens than shipping raw OOXML bytes would be even if Bedrock accepted them. +# +# Uploads are gated to keep this reachable: `.pptx` is in ALLOWED_MIME_TYPES +# above, and the frontend allowlist in `file-upload.service.ts` must stay in +# sync โ€” drift there is what made .pptx un-uploadable while the create-deck +# tool's own error text told users to "upload a .pptx template first". + +PRESENTATION_MIME_TYPES = frozenset({ + "application/vnd.openxmlformats-officedocument.presentationml.presentation", +}) + +PRESENTATION_EXTENSIONS = frozenset({".pptx"}) + + +def is_presentation_file(filename: str, mime_type: str) -> bool: + """Return True when the file should be handled by the PowerPoint tools + rather than sent inline as a Bedrock document block. + """ + if mime_type and mime_type.lower() in PRESENTATION_MIME_TYPES: + return True + if filename: + lower = filename.lower() + for ext in PRESENTATION_EXTENSIONS: + if lower.endswith(ext): + return True + return False + + # Bedrock's /ConverseStream imposes a 4.5MB hard limit on each document # content block's *internal* representation. Non-tabular formats (PDF, docx, # txt, md) don't inflate much, but we leave margin for per-request overhead diff --git a/backend/src/apis/shared/oauth/models.py b/backend/src/apis/shared/oauth/models.py index bab82469f..7809e09cc 100644 --- a/backend/src/apis/shared/oauth/models.py +++ b/backend/src/apis/shared/oauth/models.py @@ -365,6 +365,18 @@ class OAuthRequiredEvent(BaseModel): consent popup at `authorizationUrl`, and on completion POSTs an interrupt response carrying `interruptId` back to `/invocations`. The backend resumes the same turn โ€” no retype, no replay. + + `interruptId` is omitted for the *pre-flight* flavor, emitted when an + OAuth-gated MCP server refused `tools/list` and the tool never made it + into the registry (see `ExternalMCPIntegration._recover_oauth_preflight`). + There is no paused turn to resume in that case โ€” the turn ran to + completion without the tool โ€” so the frontend must show the Connect + affordance without auto-resuming. Sending a synthetic id instead would + be worse than omitting it: the resume guard in + `inference_api/chat/routes.py` rejects unknown ids with a 400, so the + user would complete consent and then be shown an error. The SPA already + guards its resume call on `request.interruptId` being present, and + dedupes concurrent prompts by `providerId`. """ model_config = ConfigDict(populate_by_name=True) @@ -372,7 +384,7 @@ class OAuthRequiredEvent(BaseModel): type: str = "oauth_required" provider_id: str = Field(..., alias="providerId") authorization_url: str = Field(..., alias="authorizationUrl") - interrupt_id: str = Field(..., alias="interruptId") + interrupt_id: Optional[str] = Field(None, alias="interruptId") def to_sse_format(self) -> str: import json diff --git a/backend/src/apis/shared/oauth/token_resolution.py b/backend/src/apis/shared/oauth/token_resolution.py new file mode 100644 index 000000000..885b85a92 --- /dev/null +++ b/backend/src/apis/shared/oauth/token_resolution.py @@ -0,0 +1,115 @@ +"""Shared "token or consent URL?" query against AgentCore Identity. + +Two places need to ask AgentCore the same question about a +(user, OAuth provider) pair: + + * ``OAuthConsentHook`` โ€” at tool-call time, to gate a tool the model is + about to run. + * ``ExternalMCPIntegration.load_external_tools`` โ€” at agent-build time, + when an OAuth-gated MCP server refuses the pre-flight ``tools/list`` + because the in-process token cache is cold. + +Both must ask with the **same** ``scopes`` and ``customParameters``. +AgentCore folds both into the token-vault key, so querying with a +different set looks up a *different* vault entry and hands back a consent +URL for a user who has already authorized โ€” an infinite "please connect" +loop for a connector that is in fact connected. Reading the provider +record in one place is what keeps the two callers in agreement; see +``docs`` on the connector's admin-configured Custom OAuth Parameters. + +The hook keeps its own injected-lookup structure (it memoizes per turn and +is unit-tested with fakes); this module is the path used by callers that +have no hook instance to borrow. +""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Optional, TypedDict + +from apis.shared.oauth.agentcore_identity import ( + CallbackUrlUnavailableError, + WorkloadTokenUnavailableError, + custom_parameters_for, + get_agentcore_identity_client, +) + +logger = logging.getLogger(__name__) + + +class TokenOrConsent(TypedDict): + """Exactly one of these is populated by AgentCore Identity.""" + + token: Optional[str] + url: Optional[str] + + +async def resolve_token_or_consent_url( + provider_id: str, + user_id: str, + *, + force_authentication: bool = False, +) -> Optional[TokenOrConsent]: + """Ask AgentCore Identity for a vaulted token, else a consent URL. + + Returns ``{"token": ..., "url": ...}`` with exactly one side populated, + or ``None`` on a hard error (missing workload token / callback URL / + unexpected failure). ``None`` means "couldn't ask" โ€” it is NOT the same + as "user must consent", and callers must not prompt on it. + + Args: + provider_id: Credential provider name registered with AgentCore + Identity, as stored on the tool's ``requires_oauth_provider``. + user_id: The platform user the token is federated for. + force_authentication: Bypass the vault and force a fresh consent โ€” + used only for an explicit user disconnect. + """ + from apis.shared.oauth.provider_repository import get_provider_repository + + try: + provider = await get_provider_repository().get_provider(provider_id) + except asyncio.CancelledError: + raise + except Exception: + logger.exception( + "Failed to read OAuth provider record for provider=%s", provider_id + ) + return None + + # A missing provider record is a misconfiguration, not a consent gap. + # Returning None keeps the caller from prompting for a connector that + # the admin has since deleted. + if provider is None: + logger.warning( + "No OAuth provider record for provider=%s; cannot resolve a token", + provider_id, + ) + return None + + identity_client = get_agentcore_identity_client() + try: + result = await identity_client.get_token_for_user( + provider_name=provider_id, + scopes=provider.scopes or [], + user_id=user_id, + force_authentication=force_authentication, + custom_parameters=custom_parameters_for(provider.custom_parameters), + ) + except WorkloadTokenUnavailableError: + logger.error( + "No workload token on context for provider=%s โ€” " + "AgentCoreContextMiddleware may be misconfigured", + provider_id, + ) + return None + except CallbackUrlUnavailableError as err: + logger.error("No OAuth2 callback URL for provider=%s: %s", provider_id, err) + return None + except asyncio.CancelledError: + raise + except Exception: + logger.exception("Failed to fetch OAuth token for provider=%s", provider_id) + return None + + return {"token": result.access_token, "url": result.authorization_url} diff --git a/backend/src/apis/shared/sessions/metadata.py b/backend/src/apis/shared/sessions/metadata.py index c4162edcc..c2033af48 100644 --- a/backend/src/apis/shared/sessions/metadata.py +++ b/backend/src/apis/shared/sessions/metadata.py @@ -2843,6 +2843,20 @@ async def clear_truncated_turn(session_id: str, user_id: str) -> None: logger.error("Failed to clear truncated_turn: %s", e, exc_info=True) +# Interrupt-reason precedence, strongest first. Rank = how much the reason +# actually tells us, which is why the client-attested ones outrank the +# server's fallback: `connection_lost` is stamped whenever a stream is torn +# down and nothing said why, so a refresh, a dead socket and a platform-side +# idle timeout are indistinguishable under it. A reason may only overwrite a +# same-or-weaker one (see `set_interrupted_turn`). +_REASON_RANK = { + "unknown": 0, + "connection_lost": 1, + "navigated_away": 2, + "user_stopped": 3, +} + + async def set_interrupted_turn( session_id: str, user_id: str, @@ -2852,22 +2866,34 @@ async def set_interrupted_turn( """Mark that the last turn was interrupted before completion. Interruptions come from two racing sources that write the same session - record: the client stop signal (app-api ``POST /sessions/{id}/interrupt``, - ``reason="user_stopped"``) and the stream cancellation backstop - (inference-api, ``reason="connection_lost"`` fallback). ``user_stopped`` - is the stronger signal, so a ``user_stopped`` write is unconditional - while a fallback write is guarded by a condition so it can never - downgrade an already-recorded ``user_stopped`` โ€” whichever source lands - first, the final reason is correct. Idempotent. Best-effort: a write - failure logs but never breaks the live flow. No-op when the session - record is missing or the table env var is unset. + record: the client signal (app-api ``POST /sessions/{id}/interrupt`` โ€” + ``user_stopped`` from the Stop button, ``navigated_away`` from the + page-lifecycle handler) and the stream cancellation backstop + (inference-api, ``connection_lost`` fallback). + + Precedence is by ``_REASON_RANK``: a write only lands if no *stronger* + reason is already recorded, enforced as a DynamoDB condition against the + pre-update item, so the outcome is correct regardless of which source + wins the race. The ranking is by how much the reason actually tells us โ€” + a client-attested reason outranks the server's fallback, which is + literally "the stream died and nothing told us why". + + Note this used to protect only ``user_stopped``, which meant the + ``connection_lost`` backstop could overwrite any other reason it raced. + That was harmless while ``user_stopped`` was the only client reason; + with ``navigated_away`` it would silently erase the attribution this + exists to capture. + + Idempotent. Best-effort: a write failure logs but never breaks the live + flow. No-op when the session record is missing or the table env var is + unset. """ sessions_metadata_table = os.environ.get("DYNAMODB_SESSIONS_METADATA_TABLE_NAME") if not sessions_metadata_table: logger.warning("DYNAMODB_SESSIONS_METADATA_TABLE_NAME not set; skipping interrupted_turn persistence") return - if reason not in ("user_stopped", "connection_lost", "unknown"): + if reason not in _REASON_RANK: reason = "unknown" try: @@ -2904,13 +2930,21 @@ async def set_interrupted_turn( }, } - # A fallback (non-user_stopped) write must not clobber a stronger - # user_stopped reason the beacon may have already landed. The - # condition is evaluated against the pre-update item, so this is - # race-safe regardless of which source writes first. - if reason != "user_stopped": - update_kwargs["ConditionExpression"] = "attribute_not_exists(#ltr) OR #ltr <> :user_stopped" - update_kwargs["ExpressionAttributeValues"][":user_stopped"] = "user_stopped" + # A write must not clobber a reason that says more than this one + # does. Guard against every strictly-stronger reason; evaluated + # against the pre-update item, so it is race-safe regardless of + # which source writes first. The strongest reason has no stronger + # peers, so it writes unconditionally. + stronger = [r for r, rank in _REASON_RANK.items() if rank > _REASON_RANK[reason]] + if stronger: + clauses = [] + for i, r in enumerate(stronger): + placeholder = f":stronger{i}" + clauses.append(f"#ltr <> {placeholder}") + update_kwargs["ExpressionAttributeValues"][placeholder] = r + update_kwargs["ConditionExpression"] = ( + "attribute_not_exists(#ltr) OR (" + " AND ".join(clauses) + ")" + ) try: table.update_item(**update_kwargs) diff --git a/backend/src/apis/shared/sessions/models.py b/backend/src/apis/shared/sessions/models.py index 105ccb0cc..434dd9616 100644 --- a/backend/src/apis/shared/sessions/models.py +++ b/backend/src/apis/shared/sessions/models.py @@ -241,10 +241,17 @@ class SessionMetadata(BaseModel): alias="lastTurnInterrupted", description="True when the last turn was interrupted before completion (user Stop, refresh, or dropped connection). Lets a reload show the 'response interrupted' state. Cleared at the start of any new (non-interrupt-resume) turn", ) - last_turn_interrupt_reason: Optional[Literal["user_stopped", "connection_lost", "unknown"]] = Field( + last_turn_interrupt_reason: Optional[ + Literal["user_stopped", "navigated_away", "connection_lost", "unknown"] + ] = Field( default=None, alias="lastTurnInterruptReason", - description="Why the last turn was interrupted. 'user_stopped' (deliberate Stop, from the client beacon) wins over the 'connection_lost' cancellation fallback", + description=( + "Why the last turn was interrupted, strongest client-attested reason first: " + "'user_stopped' (deliberate Stop) > 'navigated_away' (page hidden/unloaded) > " + "'connection_lost' (the server-side cancellation fallback) > 'unknown'. " + "A weaker reason can never overwrite a stronger one โ€” see set_interrupted_turn" + ), ) last_turn_interrupted_at: Optional[str] = Field( default=None, @@ -321,14 +328,29 @@ class UpdateSessionMetadataRequest(BaseModel): class SessionInterruptRequest(BaseModel): """Request body for the client stop signal (POST /sessions/{id}/interrupt). - Only `user_stopped` is accepted from the client โ€” it is the one reason - that requires user attestation. `connection_lost` is never client-sent; - it is inferred server-side by the stream-cancellation backstop and would - otherwise let a client downgrade a deliberate stop. + Only client-*attested* reasons are accepted here โ€” the ones the browser + is the sole witness to: + + * `user_stopped` โ€” the Stop button. Deliberate rejection of the + in-flight response. + * `navigated_away` โ€” the page was hidden or unloaded (refresh, tab + close, navigation) while a turn was streaming. + NOT a rejection: the user left, they didn't say + "stop". Sent from a `pagehide` handler. + + `connection_lost` is never client-sent. It is the server-side + cancellation backstop's fallback โ€” literally "the stream died and nothing + told us why" โ€” and accepting it from a client would let a caller + downgrade an attested reason. + + The distinction exists because `connection_lost` is otherwise + unattributable: a refresh, a dropped socket, and a platform-side idle + timeout all reach the container as an identical cancellation. Labelling + departures at the source is what makes the remainder diagnosable. """ - reason: Literal["user_stopped"] = Field( - description="Interruption reason. Only the deliberate Stop is client-attested", + reason: Literal["user_stopped", "navigated_away"] = Field( + description="Interruption reason. Only client-attested reasons are accepted", ) @@ -380,7 +402,7 @@ class SessionMetadataResponse(BaseModel): last_turn_interrupt_reason: Optional[str] = Field( default=None, alias="lastTurnInterruptReason", - description="Why the last turn was interrupted: 'user_stopped' (deliberate Stop) or 'connection_lost' (refresh / dropped connection). Drives whether a 'Continue' affordance is offered on reload", + description="Why the last turn was interrupted: 'user_stopped' (deliberate Stop), 'navigated_away' (page hidden/unloaded mid-turn), or 'connection_lost' (the unattributable server-side fallback). Only 'user_stopped' suppresses the 'Continue' affordance on reload", ) last_turn_interrupted_at: Optional[str] = Field( default=None, diff --git a/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py b/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py index 19db6b5f0..360bd3001 100644 --- a/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py +++ b/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py @@ -12,15 +12,57 @@ from types import SimpleNamespace from unittest.mock import AsyncMock, patch +import httpx import pytest +from strands.types.exceptions import ( + MCPClientInitializationError, + ToolProviderException, +) from agents.main_agent.integrations.external_mcp_client import ( ExternalMCPIntegration, + _is_auth_failure, detect_aws_service_from_url, extract_region_from_url, ) +def _wrapped_preflight_failure(inner: BaseException) -> Exception: + """Rebuild the exception `client.load_tools()` actually raises. + + Strands opens the MCP session on a background thread, so the real cause + surfaces inside an anyio `ExceptionGroup`, wrapped by + `MCPClientInitializationError` (raised by `start()`), wrapped again by + `ToolProviderException` (raised by `load_tools()`). Neither wrapper's + message carries the status, so a classifier that only looks at the + exception it caught sees nothing โ€” which is the whole point of these + tests using the real shape instead of a bare `RuntimeError`. + """ + group = ExceptionGroup("unhandled errors in a TaskGroup (1 sub-exception)", [inner]) + init_exc = MCPClientInitializationError( + f"the client initialization failed: {group}" + ) + init_exc.__cause__ = group + outer = ToolProviderException(f"Failed to start MCP client: {init_exc}") + outer.__cause__ = init_exc + return outer + + +def _http_status_error( + status: int, message: str = "the server rejected the request" +) -> httpx.HTTPStatusError: + """The error `response.raise_for_status()` raises inside the MCP transport. + + `message` defaults to text with no status in it, so tests using this + exercise the *structural* check (`.response.status_code`) rather than + accidentally passing on httpx's usual "Client error '401 Unauthorized'" + wording. + """ + request = httpx.Request("POST", "https://api.example.com/mcp") + response = httpx.Response(status, request=request) + return httpx.HTTPStatusError(message, request=request, response=response) + + class TestExtractRegionFromUrl: """Tests for extract_region_from_url region extraction.""" @@ -503,3 +545,360 @@ def test_missing_returns_none(self): integration = ExternalMCPIntegration() assert integration.get_client("canvas::courses", "alice") is None assert integration.get_client("canvas") is None + + +def _status_code_error(status: int) -> Exception: + """An exception carrying the status directly, with no httpx `.response`.""" + exc = RuntimeError("request rejected") + exc.status_code = status + return exc + + +class TestIsAuthFailure: + """Only a refusal to authorize can be fixed by consenting. Everything + else โ€” server down, host doesn't resolve, request timed out โ€” must not be + read as a consent gap, or an unconsented user is asked to connect a + server that isn't there, on every turn, forever.""" + + def test_wrapped_401_is_an_auth_failure(self): + """The status lives three wrappers down, in an ExceptionGroup, and + never appears in the outermost message.""" + exc = _wrapped_preflight_failure(_http_status_error(401)) + + assert "401" not in str(exc) + assert _is_auth_failure(exc) is True + + def test_wrapped_403_is_an_auth_failure(self): + assert _is_auth_failure(_wrapped_preflight_failure(_http_status_error(403))) + + def test_wrapped_connect_error_is_not_an_auth_failure(self): + """The dev `canvas_faculty` case: `serverUrl` points at a localhost + port that the AgentCore Runtime container cannot reach.""" + exc = _wrapped_preflight_failure( + httpx.ConnectError("All connection attempts failed") + ) + + assert _is_auth_failure(exc) is False + + def test_wrapped_timeout_is_not_an_auth_failure(self): + exc = _wrapped_preflight_failure(httpx.ConnectTimeout("timed out")) + + assert _is_auth_failure(exc) is False + + def test_dns_failure_is_not_an_auth_failure(self): + exc = _wrapped_preflight_failure( + httpx.ConnectError("[Errno 8] nodename nor servname provided") + ) + + assert _is_auth_failure(exc) is False + + def test_server_error_is_not_an_auth_failure(self): + """A 500 is the server's problem, not the user's authorization.""" + exc = _wrapped_preflight_failure(_http_status_error(500)) + + assert _is_auth_failure(exc) is False + + def test_plain_status_code_attribute_is_an_auth_failure(self): + """Not every client wraps the response; some carry the status directly.""" + assert _is_auth_failure(_status_code_error(403)) is True + + def test_message_only_unauthorized_is_an_auth_failure(self): + """A server that reports the refusal as protocol text rather than an + HTTP error still gets the user a consent prompt.""" + assert _is_auth_failure(RuntimeError("Unauthorized: missing bearer token")) + + def test_digits_inside_a_url_do_not_read_as_a_status(self): + """The text fallback must not fire on a port or path that merely + contains the digits โ€” that would resurrect the bug it guards.""" + assert not _is_auth_failure( + httpx.ConnectError("connection to localhost:8403 failed") + ) + assert not _is_auth_failure(RuntimeError("cannot reach https://x/v1/4010/mcp")) + + def test_self_referential_chain_terminates(self): + """`__context__` cycles are possible when exceptions are re-raised; + the walk must not spin.""" + first = RuntimeError("boom") + second = RuntimeError("bang") + first.__context__ = second + second.__context__ = first + + assert _is_auth_failure(first) is False + + +class TestOAuthPreflightRecovery: + """An OAuth-gated MCP server that requires auth even for `tools/list` + (GitHub's does) 401s whenever the in-process token cache is cold โ€” which + it always is on a fresh microVM. + + Before `_recover_oauth_preflight` the tool was dropped, and since + `OAuthConsentHook` only runs `BeforeToolCall` for *registered* tools, + nothing ever warmed the cache: the tool stayed missing for the life of + the process even for a user whose token sat in the AgentCore vault the + whole time, and the user was never told why. + """ + + PROVIDER = "github-oauth" + + @staticmethod + def _oauth_tool(tool_id="github_issues"): + return SimpleNamespace( + tool_id=tool_id, + protocol="mcp_external", + mcp_config=SimpleNamespace( + server_url="https://api.githubcopilot.com/mcp/x/issues", + approval_required_names=lambda: set(), + ), + forward_auth_token=False, + requires_oauth_provider=TestOAuthPreflightRecovery.PROVIDER, + updated_at=datetime(2025, 1, 1, tzinfo=timezone.utc), + ) + + @staticmethod + def _patches(client, repo, resolved): + """Common patch stack; `resolved` is the vault's answer.""" + return ( + patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ), + patch( + "agents.main_agent.integrations.external_mcp_client." + "create_external_mcp_client", + return_value=client, + ), + patch( + "apis.shared.oauth.token_resolution.resolve_token_or_consent_url", + AsyncMock(return_value=resolved), + ), + ) + + @pytest.mark.asyncio + async def test_warms_vault_token_and_registers_tool(self): + """Consented user: the vault has a token, so the retry succeeds and + the tool comes back instead of vanishing for the process lifetime.""" + from agents.main_agent.integrations import oauth_token_cache + + integration = ExternalMCPIntegration() + repo = SimpleNamespace(get_tool=AsyncMock(return_value=self._oauth_tool())) + # Fails cold, succeeds once a token is in the cache. + client = SimpleNamespace( + load_tools=AsyncMock(side_effect=[RuntimeError("401 Unauthorized"), []]) + ) + resolved = {"token": "vault-token", "url": None} + + oauth_token_cache.clear_user_provider("alice", self.PROVIDER) + p1, p2, p3 = self._patches(client, repo, resolved) + try: + with p1, p2, p3: + result = await integration.load_external_tools( + ["github_issues"], user_id="alice" + ) + + assert result == [client] + assert client.load_tools.await_count == 2 + # Cache warmed so the client's lazy token provider can use it. + assert oauth_token_cache.get("alice", self.PROVIDER) == "vault-token" + # Nothing to prompt for โ€” the user is already connected. + assert integration.take_pending_consents("alice") == {} + finally: + oauth_token_cache.clear_user_provider("alice", self.PROVIDER) + + @pytest.mark.asyncio + async def test_records_pending_consent_when_vault_returns_url(self): + """Unconsented user: AgentCore hands back an authorization URL, so + the drop is recorded for the turn to surface as `oauth_required`.""" + integration = ExternalMCPIntegration() + repo = SimpleNamespace(get_tool=AsyncMock(return_value=self._oauth_tool())) + client = SimpleNamespace( + load_tools=AsyncMock(side_effect=RuntimeError("401 Unauthorized")) + ) + resolved = {"token": None, "url": "https://consent.example/authorize"} + + p1, p2, p3 = self._patches(client, repo, resolved) + with p1, p2, p3: + result = await integration.load_external_tools( + ["github_issues"], user_id="alice" + ) + + assert result == [] + assert "github_issues" not in integration.clients + assert integration.take_pending_consents("alice") == { + self.PROVIDER: "https://consent.example/authorize" + } + + @pytest.mark.asyncio + async def test_no_prompt_when_vault_cannot_be_reached(self): + """A hard error is "couldn't ask", NOT "user must consent". Prompting + here would nag a connected user every turn the vault is unhappy.""" + integration = ExternalMCPIntegration() + repo = SimpleNamespace(get_tool=AsyncMock(return_value=self._oauth_tool())) + # 401 so the pre-flight really does reach the vault โ€” the vault + # itself is what fails here. + client = SimpleNamespace( + load_tools=AsyncMock(side_effect=RuntimeError("401 Unauthorized")) + ) + + p1, p2, p3 = self._patches(client, repo, None) + with p1, p2, p3: + result = await integration.load_external_tools( + ["github_issues"], user_id="alice" + ) + + assert result == [] + assert integration.take_pending_consents("alice") == {} + + @pytest.mark.asyncio + async def test_unreachable_server_never_prompts_for_consent(self): + """The dev `canvas_faculty` regression: an OAuth-gated server whose + URL the runtime cannot reach failed pre-flight with a ConnectError, + the vault correctly answered "no token, here is an authorization + URL", and the user was shown a connect prompt on every turn that + completing consent could never satisfy โ€” the server simply is not + there. Classify before asking.""" + integration = ExternalMCPIntegration() + repo = SimpleNamespace( + get_tool=AsyncMock(return_value=self._oauth_tool(tool_id="canvas_faculty")) + ) + client = SimpleNamespace( + load_tools=AsyncMock( + side_effect=_wrapped_preflight_failure( + httpx.ConnectError("All connection attempts failed") + ) + ) + ) + resolve_mock = AsyncMock( + return_value={"token": None, "url": "https://consent.example/authorize"} + ) + + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ), patch( + "agents.main_agent.integrations.external_mcp_client." + "create_external_mcp_client", + return_value=client, + ), patch( + "apis.shared.oauth.token_resolution.resolve_token_or_consent_url", + resolve_mock, + ): + result = await integration.load_external_tools( + ["canvas_faculty"], user_id="alice" + ) + + assert result == [] + # Never even asked โ€” the vault would have said "not consented" and + # that answer is meaningless for a server that isn't listening. + resolve_mock.assert_not_awaited() + assert integration.take_pending_consents("alice") == {} + + @pytest.mark.asyncio + async def test_wrapped_401_still_records_consent(self): + """The real production shape, not a bare `RuntimeError`: a 401 buried + under Strands' two wrappers and an anyio ExceptionGroup must still + reach the vault, or classifying would undo the recovery it guards.""" + integration = ExternalMCPIntegration() + repo = SimpleNamespace(get_tool=AsyncMock(return_value=self._oauth_tool())) + client = SimpleNamespace( + load_tools=AsyncMock( + side_effect=_wrapped_preflight_failure(_http_status_error(401)) + ) + ) + resolved = {"token": None, "url": "https://consent.example/authorize"} + + p1, p2, p3 = self._patches(client, repo, resolved) + with p1, p2, p3: + result = await integration.load_external_tools( + ["github_issues"], user_id="alice" + ) + + assert result == [] + assert integration.take_pending_consents("alice") == { + self.PROVIDER: "https://consent.example/authorize" + } + + @pytest.mark.asyncio + async def test_skips_vault_when_cached_token_already_present(self): + """A warm token that still failed is an expiry case owned by the + hook's AfterToolCall 401 handler โ€” re-asking the vault without + force_authentication would just return the same token.""" + from agents.main_agent.integrations import oauth_token_cache + + integration = ExternalMCPIntegration() + repo = SimpleNamespace(get_tool=AsyncMock(return_value=self._oauth_tool())) + client = SimpleNamespace( + load_tools=AsyncMock(side_effect=RuntimeError("500 Server Error")) + ) + resolve_mock = AsyncMock(return_value={"token": "t", "url": None}) + + oauth_token_cache.set("alice", self.PROVIDER, "already-warm") + try: + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ), patch( + "agents.main_agent.integrations.external_mcp_client." + "create_external_mcp_client", + return_value=client, + ), patch( + "apis.shared.oauth.token_resolution.resolve_token_or_consent_url", + resolve_mock, + ): + result = await integration.load_external_tools( + ["github_issues"], user_id="alice" + ) + + assert result == [] + resolve_mock.assert_not_awaited() + assert integration.take_pending_consents("alice") == {} + finally: + oauth_token_cache.clear_user_provider("alice", self.PROVIDER) + + @pytest.mark.asyncio + async def test_non_oauth_tool_failure_never_consults_the_vault(self): + """Unchanged behaviour for a plain unreachable server.""" + integration = ExternalMCPIntegration() + repo = SimpleNamespace( + get_tool=AsyncMock( + return_value=_fake_tool(datetime(2025, 1, 1, tzinfo=timezone.utc)) + ) + ) + client = SimpleNamespace( + load_tools=AsyncMock(side_effect=RuntimeError("connection refused")) + ) + resolve_mock = AsyncMock() + + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ), patch( + "agents.main_agent.integrations.external_mcp_client." + "create_external_mcp_client", + return_value=client, + ), patch( + "apis.shared.oauth.token_resolution.resolve_token_or_consent_url", + resolve_mock, + ): + result = await integration.load_external_tools(["gmail"], user_id="alice") + + assert result == [] + resolve_mock.assert_not_awaited() + + def test_take_pending_consents_drains_and_is_per_user(self): + """Draining stops a stale prompt re-firing on an agent-cache hit, + and one user's consent must never leak into another's stream.""" + integration = ExternalMCPIntegration() + integration._pending_consents = { + "alice": {self.PROVIDER: "https://consent.example/a"}, + "bob": {self.PROVIDER: "https://consent.example/b"}, + } + + assert integration.take_pending_consents("alice") == { + self.PROVIDER: "https://consent.example/a" + } + assert integration.take_pending_consents("alice") == {} + # Bob's entry is untouched by Alice's drain. + assert integration.take_pending_consents("bob") == { + self.PROVIDER: "https://consent.example/b" + } + assert integration.take_pending_consents("carol") == {} diff --git a/backend/tests/agents/main_agent/multimodal/test_attachment_marker.py b/backend/tests/agents/main_agent/multimodal/test_attachment_marker.py new file mode 100644 index 000000000..dcb81ca3b --- /dev/null +++ b/backend/tests/agents/main_agent/multimodal/test_attachment_marker.py @@ -0,0 +1,109 @@ +"""The `[Attached files: โ€ฆ]` marker must name every attachment, not just inline ones. + +The marker is the ONLY link between an uploaded file and the message it was +attached to once a session is reloaded. The SPA renders attachment cards from +a `fileAttachment` content block that it builds client-side at send time and +that is never persisted; on reload it reconstructs those blocks by parsing +this marker out of the message text and matching the names against +`GET /files?sessionId=โ€ฆ` (``restoreFileAttachments`` in message-map.service.ts). + +So a filename missing from the marker is not a cosmetic problem โ€” that file's +card silently disappears from the conversation on refresh, while the file +itself is still perfectly present in the session. + +That is what happened when the presentation carve-out shipped: the marker was +derived from the inline set, decks are deliberately not in the inline set, and +a lone .pptx even took ``build_prompt``'s "no files โ†’ return the bare message" +early path, so it left no trace at all. Spreadsheets had the same gap from the +tabular carve-out. + +Two invariants hold this together, and both are load-bearing: + +* the marker names diverted attachments as well as inline ones; +* the marker stays at the very END of the text, because the SPA's + ``ATTACHED_FILES_PATTERN`` is ``$``-anchored โ€” anything appended after it + makes the regex miss and the cards vanish just as completely. +""" + +import re + +from agents.main_agent.multimodal.prompt_builder import PromptBuilder + +# Mirrors ATTACHED_FILES_PATTERN in +# frontend/ai.client/src/app/session/services/session/message-map.service.ts +SPA_MARKER_PATTERN = re.compile(r"\n\n\[Attached files: ([^\]]+)\]$") + + +def _text_of(result): + """The prompt's text, whether build_prompt returned a str or blocks.""" + return result if isinstance(result, str) else result[0]["text"] + + +class TestMarkerNamesDivertedAttachments: + def test_lone_diverted_deck_still_produces_a_marker(self): + # The regression: nothing inline, so the old code returned the bare + # message and the deck vanished from restored history. + result = PromptBuilder().build_prompt( + "Summarize this", files=None, attachment_names=["deck.pptx"] + ) + assert _text_of(result) == "Summarize this\n\n[Attached files: deck.pptx]" + + def test_marker_covers_inline_and_diverted_together(self, sample_files): + inline = sample_files[0] + result = PromptBuilder().build_prompt( + "Compare these", + files=[inline], + attachment_names=[inline.filename, "deck.pptx", "data.xlsx"], + ) + names = SPA_MARKER_PATTERN.search(_text_of(result)).group(1) + assert names == f"{inline.filename}, deck.pptx, data.xlsx" + + def test_diverted_names_do_not_become_content_blocks(self, sample_files): + # The marker naming a deck must not smuggle it into the blocks โ€” a + # pptx document block is a Bedrock ValidationException. + inline = sample_files[0] + result = PromptBuilder().build_prompt( + "Compare these", + files=[inline], + attachment_names=[inline.filename, "deck.pptx"], + ) + assert isinstance(result, list) + # One text block + exactly one block for the single inline file. + assert len(result) == 2 + assert "deck.pptx" not in str(result[1]) + + +class TestMarkerIsSpaParseable: + def test_marker_is_last_so_the_anchored_regex_matches(self): + result = PromptBuilder().build_prompt( + "Question here", files=None, attachment_names=["a.pptx", "b.xlsx"] + ) + match = SPA_MARKER_PATTERN.search(_text_of(result)) + assert match is not None + assert match.group(1).split(", ") == ["a.pptx", "b.xlsx"] + + def test_stripping_the_marker_leaves_the_users_text(self): + # The SPA removes the marker before display; what's left must be the + # message, not a fragment of it. + result = PromptBuilder().build_prompt( + "What is in here?", files=None, attachment_names=["deck.pptx"] + ) + assert SPA_MARKER_PATTERN.sub("", _text_of(result)) == "What is in here?" + + +class TestBackwardCompatibility: + def test_omitting_attachment_names_still_derives_them_from_files(self, sample_files): + # Existing callers pass positionally; behaviour must not shift. + inline = sample_files[0] + result = PromptBuilder().build_prompt("Describe this", [inline]) + assert SPA_MARKER_PATTERN.search(_text_of(result)).group(1) == inline.filename + + def test_no_files_and_no_names_returns_the_bare_message(self): + result = PromptBuilder().build_prompt("Just talking") + assert result == "Just talking" + + def test_empty_names_list_adds_no_marker(self): + result = PromptBuilder().build_prompt( + "Just talking", files=None, attachment_names=[] + ) + assert result == "Just talking" diff --git a/backend/tests/agents/main_agent/streaming/test_preflight_consent_events.py b/backend/tests/agents/main_agent/streaming/test_preflight_consent_events.py new file mode 100644 index 000000000..2749fa114 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_preflight_consent_events.py @@ -0,0 +1,120 @@ +"""`oauth_required` for OAuth-gated MCP tools dropped at agent-build time. + +When an OAuth-gated MCP server refuses the pre-flight `tools/list`, the tool +never enters the registry โ€” so `OAuthConsentHook`, which is a +`BeforeToolCall` hook, can never fire for it and the user is never told the +tool is missing. `_extract_preflight_consent_events` closes that gap by +draining the integration's recorded consents at the end of the turn. + +These events carry NO `interruptId`: nothing is paused, so there is nothing +to resume. Sending a synthetic id instead would be actively worse โ€” the +resume guard in `inference_api/chat/routes.py` rejects unknown ids with a +400, so the user would complete consent and then be shown an error. +""" + +from __future__ import annotations + +import json +from types import SimpleNamespace +from unittest.mock import patch + +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + + +def _parse(sse: str) -> dict: + """Pull the JSON payload out of an SSE frame.""" + data_line = next( + line for line in sse.splitlines() if line.startswith("data: ") + ) + return json.loads(data_line[len("data: ") :]) + + +def _coordinator() -> StreamCoordinator: + return StreamCoordinator.__new__(StreamCoordinator) + + +def _integration(pending: dict[str, dict[str, str]]): + store = dict(pending) + return SimpleNamespace(take_pending_consents=lambda uid: store.pop(uid, {})) + + +class TestPreflightConsentEvents: + def test_emits_oauth_required_without_interrupt_id(self): + coordinator = _coordinator() + integration = _integration( + {"alice": {"github-oauth": "https://consent.example/authorize"}} + ) + + with patch( + "agents.main_agent.integrations.external_mcp_client." + "get_external_mcp_integration", + return_value=integration, + ): + events = coordinator._extract_preflight_consent_events("alice") + + assert len(events) == 1 + assert events[0].startswith("event: oauth_required\n") + payload = _parse(events[0]) + assert payload["providerId"] == "github-oauth" + assert payload["authorizationUrl"] == "https://consent.example/authorize" + # The whole point: no resumable id, and the key must be absent + # rather than null โ€” the SPA validator rejects an empty string. + assert "interruptId" not in payload + + def test_emits_one_event_per_provider_deterministically(self): + coordinator = _coordinator() + integration = _integration( + { + "alice": { + "zeta-oauth": "https://consent.example/z", + "alpha-oauth": "https://consent.example/a", + } + } + ) + + with patch( + "agents.main_agent.integrations.external_mcp_client." + "get_external_mcp_integration", + return_value=integration, + ): + events = coordinator._extract_preflight_consent_events("alice") + + # Sorted so the frame order is stable across turns. + assert [_parse(e)["providerId"] for e in events] == [ + "alpha-oauth", + "zeta-oauth", + ] + + def test_no_events_when_nothing_pending(self): + coordinator = _coordinator() + integration = _integration({}) + + with patch( + "agents.main_agent.integrations.external_mcp_client." + "get_external_mcp_integration", + return_value=integration, + ): + assert coordinator._extract_preflight_consent_events("alice") == [] + + def test_anonymous_turn_emits_nothing(self): + """No user id means no per-user bucket to drain โ€” and draining the + wrong one would leak another user's consent into this stream.""" + coordinator = _coordinator() + assert coordinator._extract_preflight_consent_events(None) == [] + + def test_integration_failure_does_not_break_the_stream(self): + """Consent prompts are a nicety; a raise here must not kill the + turn's `done` frame.""" + coordinator = _coordinator() + boom = SimpleNamespace( + take_pending_consents=lambda uid: (_ for _ in ()).throw( + RuntimeError("boom") + ) + ) + + with patch( + "agents.main_agent.integrations.external_mcp_client." + "get_external_mcp_integration", + return_value=boom, + ): + assert coordinator._extract_preflight_consent_events("alice") == [] diff --git a/backend/tests/agents/main_agent/streaming/test_stale_interrupt_reset.py b/backend/tests/agents/main_agent/streaming/test_stale_interrupt_reset.py new file mode 100644 index 000000000..04dd23343 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_stale_interrupt_reset.py @@ -0,0 +1,273 @@ +"""Abandoning a paused turn when the user types instead of consenting. + +`OAuthConsentHook` pauses a turn by calling `event.interrupt(...)`, which +sets `_interrupt_state.activated` on the agent. The agent is cached across +turns, so if the user never completes consent and just sends a new message, +that flag is still armed โ€” and Strands' `InterruptState.resume` rejects a +plain string prompt with + + TypeError: prompt_type= | must resume from interrupt with + list of interruptResponse's + +which reached the user as a non-recoverable `stream_error`, on every +subsequent turn, for the life of the process. + +Two things have to happen together: drop the flag, and leave `agent.messages` +in a shape Bedrock accepts. Strands appends the assistant `toolUse` message +before running tools and returns on interrupt without appending the matching +`toolResult`, so the history ends on an unanswered tool call. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from agents.main_agent.streaming.stream_coordinator import ( + _drop_abandoned_turn_tail, + _is_interrupt_resume_prompt, + reset_stale_interrupt_state, +) + + +class _InterruptState: + """Stand-in for strands.interrupt.InterruptState.""" + + def __init__(self, activated: bool, interrupts: dict | None = None): + self.activated = activated + self.interrupts = interrupts or {} + self.context = {"tool_use_message": {}} + + def deactivate(self) -> None: + self.interrupts = {} + self.context = {} + self.activated = False + + +def _paused_agent(messages: list) -> SimpleNamespace: + return SimpleNamespace( + _interrupt_state=_InterruptState(True, {"i-1": object()}), + messages=messages, + ) + + +def _completed_turn() -> list: + return [ + {"role": "user", "content": [{"text": "hello"}]}, + {"role": "assistant", "content": [{"text": "hi there"}]}, + ] + + +def _abandoned_tail() -> list: + """A turn that paused on a tool call: user asked, model emitted a + toolUse, the hook interrupted before any toolResult was appended.""" + return [ + {"role": "user", "content": [{"text": "list my issues"}]}, + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t-1", "name": "github_issues"}}], + }, + ] + + +class TestIsInterruptResumePrompt: + def test_resume_payload_is_recognised(self): + prompt = [{"interruptResponse": {"interruptId": "i-1", "response": "ok"}}] + assert _is_interrupt_resume_prompt(prompt) is True + + def test_plain_string_is_not_a_resume(self): + assert _is_interrupt_resume_prompt("what's the weather") is False + + def test_multimodal_content_is_not_a_resume(self): + # A fresh turn with an attachment is a list too โ€” it must not be + # mistaken for a resume, or the stale pause survives. + prompt = [{"text": "describe this"}, {"image": {"format": "png"}}] + assert _is_interrupt_resume_prompt(prompt) is False + + def test_empty_list_is_not_a_resume(self): + # `[]` is the max_tokens "Continue" prompt. A real resume always + # carries at least one entry (`if interrupt_responses:`). + assert _is_interrupt_resume_prompt([]) is False + + def test_mixed_content_block_is_not_a_resume(self): + prompt = [{"interruptResponse": {"interruptId": "i-1"}, "text": "hi"}] + assert _is_interrupt_resume_prompt(prompt) is False + + +class TestDropAbandonedTurnTail: + def test_drops_back_to_the_last_completed_assistant_turn(self): + messages = _completed_turn() + _abandoned_tail() + dropped = _drop_abandoned_turn_tail(messages) + + assert dropped == 2 + assert messages == _completed_turn() + + def test_drops_a_multi_cycle_abandoned_turn_whole(self): + """The abandoned turn may have completed tool cycles before the one + that paused โ€” none of it survives, the turn produced no answer.""" + messages = _completed_turn() + [ + {"role": "user", "content": [{"text": "do a lot"}]}, + {"role": "assistant", "content": [{"toolUse": {"toolUseId": "t-1"}}]}, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t-1"}}]}, + {"role": "assistant", "content": [{"toolUse": {"toolUseId": "t-2"}}]}, + ] + dropped = _drop_abandoned_turn_tail(messages) + + assert dropped == 4 + assert messages == _completed_turn() + + def test_empties_history_when_the_first_turn_was_abandoned(self): + messages = _abandoned_tail() + assert _drop_abandoned_turn_tail(messages) == 2 + assert messages == [] + + def test_mutates_in_place_preserving_the_alias(self): + """The list is shared by reference between the cached agents serving + one session (#741/#750) โ€” rebinding would silently fork history.""" + messages = _completed_turn() + _abandoned_tail() + alias = messages + + _drop_abandoned_turn_tail(messages) + + assert alias is messages + assert alias == _completed_turn() + + def test_no_op_on_already_clean_history(self): + messages = _completed_turn() + assert _drop_abandoned_turn_tail(messages) == 0 + assert messages == _completed_turn() + + +class TestResetStaleInterruptState: + def test_clears_the_flag_and_the_dangling_tool_use(self): + messages = _completed_turn() + _abandoned_tail() + agent = _paused_agent(messages) + + reset_stale_interrupt_state(agent, "a brand new question") + + assert agent._interrupt_state.activated is False + assert messages == _completed_turn() + # Ends on an assistant turn, so the incoming user prompt keeps roles + # alternating, and carries no unanswered toolUse. + assert messages[-1]["role"] == "assistant" + + def test_leaves_a_genuine_resume_untouched(self): + messages = _completed_turn() + _abandoned_tail() + agent = _paused_agent(messages) + prompt = [{"interruptResponse": {"interruptId": "i-1", "response": "ok"}}] + + reset_stale_interrupt_state(agent, prompt) + + # Clearing here would destroy the very turn being resumed. + assert agent._interrupt_state.activated is True + assert len(messages) == 4 + + def test_no_op_when_no_pause_is_armed(self): + messages = _completed_turn() + agent = SimpleNamespace( + _interrupt_state=_InterruptState(False), messages=messages + ) + + reset_stale_interrupt_state(agent, "hello again") + + assert messages == _completed_turn() + + def test_continuation_clears_a_stale_pause(self): + """`[]` is a max_tokens Continue, not a resume.""" + messages = _completed_turn() + _abandoned_tail() + agent = _paused_agent(messages) + + reset_stale_interrupt_state(agent, []) + + assert agent._interrupt_state.activated is False + + def test_agent_without_interrupt_state_is_a_no_op(self): + agent = SimpleNamespace(messages=_completed_turn()) + reset_stale_interrupt_state(agent, "hi") # must not raise + + def test_deactivate_failure_leaves_history_alone(self): + """If we can't clear the flag we must not half-apply the repair โ€” + the turn will fail either way, and mangled history outlives it.""" + + class _Boom(_InterruptState): + def deactivate(self): + raise RuntimeError("boom") + + messages = _completed_turn() + _abandoned_tail() + agent = SimpleNamespace(_interrupt_state=_Boom(True), messages=messages) + + reset_stale_interrupt_state(agent, "new question") + + assert len(messages) == 4 + + +class TestStrandsContractAlignment: + """Guard against the real `InterruptState` drifting from our predicate.""" + + @pytest.mark.parametrize( + "prompt", + [ + "a string prompt", + [{"text": "multimodal"}], + ], + ) + def test_prompts_we_call_fresh_are_exactly_what_strands_rejects(self, prompt): + # Private in the SDK (`_InterruptState`) โ€” imported by its real + # name on purpose: this test exists to fail loudly if a strands + # upgrade renames or reshapes it under us. + from strands.interrupt import _InterruptState + + state = _InterruptState() + state.activate() + + assert _is_interrupt_resume_prompt(prompt) is False + with pytest.raises(TypeError, match="must resume from interrupt"): + state.resume(prompt) + + def test_a_prompt_we_call_a_resume_is_accepted_by_strands(self): + from strands.interrupt import Interrupt, _InterruptState + + state = _InterruptState() + state.interrupts = {"i-1": Interrupt(id="i-1", name="oauth:github")} + state.activate() + prompt = [{"interruptResponse": {"interruptId": "i-1", "response": "ok"}}] + + assert _is_interrupt_resume_prompt(prompt) is True + state.resume(prompt) # must not raise + assert state.interrupts["i-1"].response == "ok" + + def test_reset_makes_the_real_sdk_accept_the_next_plain_prompt(self): + """End-to-end guard on the reported bug. + + Builds a genuinely paused agent around the real `_InterruptState`, + runs the reset, then replays exactly what `stream_async` does with a + fresh prompt. Before the fix this raised the production TypeError. + """ + from strands.interrupt import Interrupt, _InterruptState + + state = _InterruptState() + state.interrupts = {"i-1": Interrupt(id="i-1", name="oauth:github-oauth")} + state.context = {"tool_use_message": {}, "tool_results": []} + state.activate() + + messages = _completed_turn() + _abandoned_tail() + agent = SimpleNamespace(_interrupt_state=state, messages=messages) + + # Pre-condition: this is the crash. + with pytest.raises(TypeError, match="must resume from interrupt"): + state.resume("a brand new question") + + reset_stale_interrupt_state(agent, "a brand new question") + + # Post-condition: the same call is now a no-op, and the history the + # prompt is about to land on is Bedrock-valid. + state.resume("a brand new question") + assert state.activated is False + assert messages[-1]["role"] == "assistant" + assert not any( + "toolUse" in block + for msg in messages + for block in (msg.get("content") or []) + if isinstance(block, dict) + ) diff --git a/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py b/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py index 31153f613..d628a1398 100644 --- a/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py +++ b/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py @@ -40,7 +40,7 @@ async def stream_response(self, **kwargs): class _PassthroughMultimodalBuilder: - def build_prompt(self, message, files): + def build_prompt(self, message, files, attachment_names=None): return message diff --git a/backend/tests/agents/main_agent/test_chat_agent_continue.py b/backend/tests/agents/main_agent/test_chat_agent_continue.py index 645d2b656..8da728fcb 100644 --- a/backend/tests/agents/main_agent/test_chat_agent_continue.py +++ b/backend/tests/agents/main_agent/test_chat_agent_continue.py @@ -25,7 +25,7 @@ async def stream_response(self, **kwargs): class _ExplodingMultimodalBuilder: """build_prompt must never be called on the continuation path.""" - def build_prompt(self, message, files): # noqa: D401 + def build_prompt(self, message, files, attachment_names=None): # noqa: D401 raise AssertionError("multimodal build_prompt called on continuation path") @@ -59,7 +59,7 @@ async def test_normal_turn_still_uses_multimodal_builder(): coordinator = _RecordingCoordinator() class _Builder: - def build_prompt(self, message, files): + def build_prompt(self, message, files, attachment_names=None): return f"built:{message}" agent = _bare_chat_agent(coordinator, _Builder()) diff --git a/backend/tests/apis/app_api/test_presentation_upload_size_cap.py b/backend/tests/apis/app_api/test_presentation_upload_size_cap.py new file mode 100644 index 000000000..5944fa31b --- /dev/null +++ b/backend/tests/apis/app_api/test_presentation_upload_size_cap.py @@ -0,0 +1,117 @@ +"""Presentations upload under a larger cap than everything else. + +The general 4MB limit is sized for Bedrock's *inline* document budget โ€” it is +the point past which a document block starts risking a ValidationException +mid-stream. A .pptx never enters that path (Bedrock's document-format enum has +no `pptx`; it routes to the PowerPoint tools instead), so the ceiling that +justifies 4MB simply does not apply to it. Corporate templates with imagery +clear 4MB routinely, which made `create_powerpoint_presentation`'s +``template_name`` argument unusable for exactly the files it exists to accept. + +The cap that *does* bind a deck is the Code Interpreter hop: +``_ci_write_bytes`` base64-encodes the whole file into a single ``writeFiles`` +``text`` field (~4/3 inflation). That field is a MaxLenString (100MB), so 25MB +of deck โ†’ ~33MB of base64 sits well inside it. + +Both size gates must agree on which cap applies. Historically they were one +constant read from two places; now that the cap depends on the file, a gate +that reads ``max_file_size`` directly rejects a deck the other gate allowed โ€” +so ``max_size_for`` is the single decision point and this pins it. +""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from apis.app_api.files.service import FileTooLargeError, FileUploadService +from apis.shared.files.models import PresignRequest + +PPTX_MIME = "application/vnd.openxmlformats-officedocument.presentationml.presentation" +GENERAL_CAP = 4 * 1024 * 1024 +PRESENTATION_CAP = 25 * 1024 * 1024 + + +@pytest.fixture +def service(): + repository = MagicMock() + repository.get_user_quota = AsyncMock( + return_value=SimpleNamespace(total_bytes=0) + ) + repository.create_file = AsyncMock() + + s3_client = MagicMock() + s3_client.generate_presigned_url.return_value = "https://example.invalid/put" + + return FileUploadService( + repository=repository, + s3_client=s3_client, + bucket_name="test-bucket", + max_file_size=GENERAL_CAP, + presentation_max_file_size=PRESENTATION_CAP, + ) + + +class TestMaxSizeFor: + def test_ordinary_documents_get_the_general_cap(self, service): + assert service.max_size_for("report.pdf", "application/pdf") == GENERAL_CAP + + def test_presentations_get_the_presentation_cap(self, service): + assert service.max_size_for("deck.pptx", PPTX_MIME) == PRESENTATION_CAP + + def test_extension_alone_is_enough(self, service): + # Some clients send octet-stream for pptx; the cap must not depend on + # the browser getting the MIME right. + assert ( + service.max_size_for("deck.pptx", "application/octet-stream") + == PRESENTATION_CAP + ) + + def test_defaults_match_the_documented_values(self): + # Constructed with no overrides โ€” these are the values the frontend + # constants mirror, and the frontend must never be the larger side. + default = FileUploadService( + repository=MagicMock(), s3_client=MagicMock(), bucket_name="b" + ) + assert default.max_file_size == GENERAL_CAP + assert default.presentation_max_file_size == PRESENTATION_CAP + + +class TestPresignSizeEnforcement: + @pytest.mark.asyncio + async def test_rejects_ordinary_document_above_general_cap(self, service): + request = PresignRequest( + sessionId="s1", + filename="big.pdf", + mimeType="application/pdf", + sizeBytes=GENERAL_CAP + 1, + ) + with pytest.raises(FileTooLargeError) as exc: + await service.request_presigned_url("u1", request) + assert exc.value.max_size == GENERAL_CAP + + @pytest.mark.asyncio + async def test_accepts_deck_above_general_cap(self, service): + # The regression this whole change exists to prevent. + request = PresignRequest( + sessionId="s1", + filename="template.pptx", + mimeType=PPTX_MIME, + sizeBytes=GENERAL_CAP + 1, + ) + response = await service.request_presigned_url("u1", request) + assert response.upload_id + + @pytest.mark.asyncio + async def test_rejects_deck_above_presentation_cap(self, service): + request = PresignRequest( + sessionId="s1", + filename="huge.pptx", + mimeType=PPTX_MIME, + sizeBytes=PRESENTATION_CAP + 1, + ) + with pytest.raises(FileTooLargeError) as exc: + await service.request_presigned_url("u1", request) + # The 400 detail is built from max_size, so the user is told 25MB โ€” + # not the 4MB that does not apply to their file. + assert exc.value.max_size == PRESENTATION_CAP diff --git a/backend/tests/apis/inference_api/test_presentation_attachment_carveout.py b/backend/tests/apis/inference_api/test_presentation_attachment_carveout.py new file mode 100644 index 000000000..87655b360 --- /dev/null +++ b/backend/tests/apis/inference_api/test_presentation_attachment_carveout.py @@ -0,0 +1,194 @@ +"""A .pptx attachment must never reach Bedrock as an inline document block. + +This is not the same kind of rule as the tabular carve-out it sits next to. +Spreadsheets are diverted as an *optimization* โ€” an xlsx would technically be +accepted inline, it just inflates past the 4.5MB internal limit and analyzes +worse than pandas would. A pptx is diverted because Bedrock's Converse +``DocumentFormat`` enum has no ``pptx`` member at all: + + pdf | csv | doc | docx | xls | xlsx | html | txt | md + +So an inline deck is an unconditional ValidationException that kills the turn, +at any size, forever โ€” not a threshold we tune. That is why +``is_presentation_file`` is checked BEFORE the size gate: routing a small deck +to the "oversized" bucket would produce a note that misdescribes why it was +skipped, and routing it to `inline` at all is simply broken. + +The upload path and this carve-out are one feature. `.pptx` is in the backend +and frontend upload allowlists only because these tools can receive it; if a +future change re-narrows either allowlist, the create-deck tool's own error +text ("Upload a .pptx template first") becomes a lie again. +""" + +import pytest + +from apis.inference_api.chat.routes import ( + _attachment_marker_names, + _build_attachment_guidance, + _partition_attachments, +) +from apis.shared.files.models import ALLOWED_MIME_TYPES, is_presentation_file + +PPTX_MIME = "application/vnd.openxmlformats-officedocument.presentationml.presentation" +XLSX_MIME = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + + +class _Attachment: + """Minimal stand-in for FileContent โ€” the partition only reads these three.""" + + def __init__(self, filename: str, content_type: str, bytes_: str = ""): + self.filename = filename + self.content_type = content_type + self.bytes = bytes_ + + +class TestIsPresentationFile: + def test_detects_by_mime_type(self): + assert is_presentation_file("anything", PPTX_MIME) is True + + def test_detects_by_extension_when_mime_is_missing(self): + # Browsers and some clients send "" or application/octet-stream for + # pptx; the extension is the fallback, same as the tabular helper. + assert is_presentation_file("deck.pptx", "") is True + assert is_presentation_file("deck.PPTX", "application/octet-stream") is True + + def test_does_not_claim_other_documents(self): + assert is_presentation_file("report.pdf", "application/pdf") is False + assert is_presentation_file("data.xlsx", XLSX_MIME) is False + + def test_legacy_ppt_is_not_claimed(self): + # .ppt (binary, pre-2007) is not in the upload allowlist and + # python-pptx cannot open it โ€” don't silently divert it. + assert is_presentation_file("old.ppt", "application/vnd.ms-powerpoint") is False + + def test_pptx_is_uploadable(self): + # The carve-out is unreachable if the upload gate rejects the file. + assert ALLOWED_MIME_TYPES.get(PPTX_MIME) == "pptx" + + +class TestPartitionAttachments: + def test_pptx_is_diverted_not_inlined(self): + deck = _Attachment("deck.pptx", PPTX_MIME) + inline, tabular, presentations, oversized = _partition_attachments([deck]) + + assert presentations == [deck] + assert inline == [] + assert tabular == [] + assert oversized == [] + + def test_tiny_pptx_still_diverted_never_oversized(self): + # The size gate must not get a vote: a 12-byte deck is still a deck, + # and "too big" would be the wrong explanation for skipping it. + deck = _Attachment("small.pptx", PPTX_MIME, bytes_="AAAA") + inline, _, presentations, oversized = _partition_attachments([deck]) + + assert presentations == [deck] + assert oversized == [] + assert inline == [] + + def test_ordinary_documents_still_inline(self): + pdf = _Attachment("report.pdf", "application/pdf", bytes_="AAAA") + _, _, presentations, _ = _partition_attachments([pdf]) + assert presentations == [] + + def test_mixed_batch_lands_in_the_right_buckets(self): + pdf = _Attachment("report.pdf", "application/pdf", bytes_="AAAA") + sheet = _Attachment("data.xlsx", XLSX_MIME) + deck = _Attachment("deck.pptx", PPTX_MIME) + + inline, tabular, presentations, oversized = _partition_attachments( + [pdf, sheet, deck] + ) + + assert inline == [pdf] + assert tabular == [sheet] + assert presentations == [deck] + assert oversized == [] + + +class TestAttachmentMarkerNames: + """Diverting a file must not erase it from the message it was attached to. + + The `[Attached files: โ€ฆ]` marker is the only link the SPA can replay on + reload โ€” see `_attachment_marker_names`. Deriving it from the inline set + is what made a diverted deck's card vanish from restored history. + """ + + def test_includes_a_diverted_deck(self): + pdf = _Attachment("report.pdf", "application/pdf") + deck = _Attachment("deck.pptx", PPTX_MIME) + assert _attachment_marker_names([pdf, deck], []) == [ + "report.pdf", + "deck.pptx", + ] + + def test_includes_a_diverted_spreadsheet(self): + sheet = _Attachment("data.xlsx", XLSX_MIME) + assert _attachment_marker_names([sheet], []) == ["data.xlsx"] + + def test_a_lone_deck_still_yields_a_name(self): + # Nothing inline at all โ€” the case that previously left no trace. + deck = _Attachment("deck.pptx", PPTX_MIME) + assert _attachment_marker_names([deck], []) == ["deck.pptx"] + + def test_excludes_oversized_files(self): + # Dropped from the turn entirely; the guidance explains their absence, + # so a card promising otherwise would be misleading. + pdf = _Attachment("report.pdf", "application/pdf") + huge = _Attachment("huge.pdf", "application/pdf") + assert _attachment_marker_names([pdf, huge], [huge]) == ["report.pdf"] + + def test_preserves_attachment_order(self): + # Order is deterministic because this text reaches the cacheable + # prefix on later turns. + files = [ + _Attachment("b.pptx", PPTX_MIME), + _Attachment("a.pdf", "application/pdf"), + _Attachment("c.xlsx", XLSX_MIME), + ] + assert _attachment_marker_names(files, []) == ["b.pptx", "a.pdf", "c.xlsx"] + + def test_no_attachments_yields_no_names(self): + assert _attachment_marker_names([], []) == [] + + +class TestAttachmentGuidance: + def test_names_the_deck_and_the_read_tool_when_enabled(self): + deck = _Attachment("deck.pptx", PPTX_MIME) + guidance = _build_attachment_guidance( + [], [deck], [], ["create_powerpoint_presentation"] + ) + + assert "`deck.pptx`" in guidance + assert "read_powerpoint_presentation" in guidance + + def test_tells_the_user_which_toggle_to_flip_when_disabled(self): + # A diverted deck with no tool to read it is a dead end unless the + # note names the toggle โ€” the file is neither inline nor reachable. + deck = _Attachment("deck.pptx", PPTX_MIME) + guidance = _build_attachment_guidance([], [deck], [], ["some_other_tool"]) + + assert "PowerPoint Presentations" in guidance + assert "read_powerpoint_presentation" not in guidance + + @pytest.mark.parametrize("enabled_tools", [None, []]) + def test_no_enabled_tools_is_treated_as_disabled(self, enabled_tools): + deck = _Attachment("deck.pptx", PPTX_MIME) + guidance = _build_attachment_guidance([], [deck], [], enabled_tools) + assert "PowerPoint Presentations" in guidance + + def test_silent_when_nothing_was_diverted(self): + assert _build_attachment_guidance([], [], [], ["create_powerpoint_presentation"]) == "" + + def test_spreadsheet_and_deck_notes_coexist(self): + # Both carve-outs can fire on one turn; neither may swallow the other. + sheet = _Attachment("data.xlsx", XLSX_MIME) + deck = _Attachment("deck.pptx", PPTX_MIME) + guidance = _build_attachment_guidance( + [sheet], [deck], [], ["analyze_spreadsheet", "create_powerpoint_presentation"] + ) + + assert "`data.xlsx`" in guidance + assert "`deck.pptx`" in guidance + assert "analyze_spreadsheet" in guidance + assert "read_powerpoint_presentation" in guidance diff --git a/backend/tests/routes/test_sessions.py b/backend/tests/routes/test_sessions.py index 105f9d1d5..30b7bc591 100644 --- a/backend/tests/routes/test_sessions.py +++ b/backend/tests/routes/test_sessions.py @@ -862,6 +862,56 @@ def test_rejects_non_client_attested_reason(self, app, make_user, authenticated_ assert resp.status_code == 422 recorder.assert_not_awaited() + def test_returns_204_and_records_navigated_away(self, app, make_user, authenticated_client): + """A departure (refresh / tab close / navigation) is client-attested + too โ€” it is the one interruption cause only the browser witnesses.""" + user = make_user() + client = authenticated_client(app, user) + + recorder = AsyncMock() + with patch( + "apis.app_api.sessions.routes.set_interrupted_turn", + recorder, + ): + resp = client.post( + "/sessions/sess-001/interrupt", + json={"reason": "navigated_away"}, + ) + + assert resp.status_code == 204 + recorder.assert_awaited_once_with( + "sess-001", + user.user_id, + reason="navigated_away", + source="client_signal", + ) + + def test_navigated_away_does_not_cancel_the_turn(self, app, make_user, authenticated_client): + """Attribution, not instruction. + + Cancelling on a departure would make every refresh kill the turn it + interrupted โ€” discarding work the reload is about to offer to + continue. Only a deliberate Stop arms the distributed cancel. + """ + user = make_user() + client = authenticated_client(app, user) + + cancel = AsyncMock(return_value=True) + with patch( + "apis.app_api.sessions.routes.set_interrupted_turn", + AsyncMock(), + ), patch( + "apis.shared.sessions.session_lease.request_session_cancel", + cancel, + ): + resp = client.post( + "/sessions/sess-001/interrupt", + json={"reason": "navigated_away"}, + ) + + assert resp.status_code == 204 + cancel.assert_not_awaited() + def test_returns_401_for_unauthenticated(self, app, unauthenticated_client): client = unauthenticated_client(app) resp = client.post( diff --git a/backend/tests/shared/test_oauth_token_resolution.py b/backend/tests/shared/test_oauth_token_resolution.py new file mode 100644 index 000000000..4a7219a46 --- /dev/null +++ b/backend/tests/shared/test_oauth_token_resolution.py @@ -0,0 +1,165 @@ +"""Tests for the shared "token or consent URL?" AgentCore query. + +The load-bearing property is the *vault-key agreement*: AgentCore folds +`scopes` and `customParameters` into the token-vault key, so this helper +must ask with exactly what the connector record says โ€” the same values +`OAuthConsentHook` sends. Drift there looks up a different vault entry and +returns a consent URL for an already-authorized user, i.e. a "please +connect" prompt that reappears no matter how often they connect. + +The other property is that a hard error is distinguishable from a consent +gap: callers prompt on a URL, never on None. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import pytest + +from apis.shared.oauth.agentcore_identity import ( + CallbackUrlUnavailableError, + WorkloadTokenUnavailableError, +) +from apis.shared.oauth.token_resolution import resolve_token_or_consent_url + + +def _provider(**overrides): + base = dict(scopes=["repo", "read:user"], custom_parameters={"prompt": "consent"}) + base.update(overrides) + return SimpleNamespace(**base) + + +def _patches(provider, identity_client): + repo = SimpleNamespace(get_provider=AsyncMock(return_value=provider)) + return ( + patch( + "apis.shared.oauth.provider_repository.get_provider_repository", + return_value=repo, + ), + patch( + "apis.shared.oauth.token_resolution.get_agentcore_identity_client", + return_value=identity_client, + ), + ) + + +class TestResolveTokenOrConsentUrl: + @pytest.mark.asyncio + async def test_returns_vaulted_token(self): + identity = SimpleNamespace( + get_token_for_user=AsyncMock( + return_value=SimpleNamespace( + access_token="tok-1", authorization_url=None + ) + ) + ) + p1, p2 = _patches(_provider(), identity) + with p1, p2: + result = await resolve_token_or_consent_url("github-oauth", "alice") + + assert result == {"token": "tok-1", "url": None} + + @pytest.mark.asyncio + async def test_sends_provider_scopes_and_custom_parameters(self): + """Vault-key agreement โ€” see the module docstring.""" + identity = SimpleNamespace( + get_token_for_user=AsyncMock( + return_value=SimpleNamespace( + access_token="tok-1", authorization_url=None + ) + ) + ) + p1, p2 = _patches(_provider(), identity) + with p1, p2: + await resolve_token_or_consent_url("github-oauth", "alice") + + kwargs = identity.get_token_for_user.await_args.kwargs + assert kwargs["provider_name"] == "github-oauth" + assert kwargs["user_id"] == "alice" + assert kwargs["scopes"] == ["repo", "read:user"] + assert kwargs["custom_parameters"] == {"prompt": "consent"} + assert kwargs["force_authentication"] is False + + @pytest.mark.asyncio + async def test_returns_consent_url_when_not_authorized(self): + identity = SimpleNamespace( + get_token_for_user=AsyncMock( + return_value=SimpleNamespace( + access_token=None, + authorization_url="https://consent.example/authorize", + ) + ) + ) + p1, p2 = _patches(_provider(), identity) + with p1, p2: + result = await resolve_token_or_consent_url("github-oauth", "alice") + + assert result == { + "token": None, + "url": "https://consent.example/authorize", + } + + @pytest.mark.asyncio + async def test_forwards_force_authentication(self): + identity = SimpleNamespace( + get_token_for_user=AsyncMock( + return_value=SimpleNamespace(access_token="t", authorization_url=None) + ) + ) + p1, p2 = _patches(_provider(), identity) + with p1, p2: + await resolve_token_or_consent_url( + "github-oauth", "alice", force_authentication=True + ) + + assert identity.get_token_for_user.await_args.kwargs[ + "force_authentication" + ] is True + + @pytest.mark.asyncio + async def test_missing_provider_record_is_not_a_consent_gap(self): + """A deleted connector must not produce a Connect prompt.""" + identity = SimpleNamespace(get_token_for_user=AsyncMock()) + p1, p2 = _patches(None, identity) + with p1, p2: + result = await resolve_token_or_consent_url("ghost", "alice") + + assert result is None + identity.get_token_for_user.assert_not_awaited() + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "exc", + [ + WorkloadTokenUnavailableError("no workload token"), + CallbackUrlUnavailableError("no callback url"), + RuntimeError("boom"), + ], + ) + async def test_hard_errors_return_none_not_a_url(self, exc): + identity = SimpleNamespace(get_token_for_user=AsyncMock(side_effect=exc)) + p1, p2 = _patches(_provider(), identity) + with p1, p2: + result = await resolve_token_or_consent_url("github-oauth", "alice") + + # None means "couldn't ask". Callers must stay silent rather than + # prompting a user who may well be connected. + assert result is None + + @pytest.mark.asyncio + async def test_provider_repository_failure_returns_none(self): + repo = SimpleNamespace(get_provider=AsyncMock(side_effect=RuntimeError("ddb"))) + identity = SimpleNamespace(get_token_for_user=AsyncMock()) + with patch( + "apis.shared.oauth.provider_repository.get_provider_repository", + return_value=repo, + ), patch( + "apis.shared.oauth.token_resolution.get_agentcore_identity_client", + return_value=identity, + ): + result = await resolve_token_or_consent_url("github-oauth", "alice") + + assert result is None + identity.get_token_for_user.assert_not_awaited() diff --git a/backend/tests/shared/test_sessions_metadata.py b/backend/tests/shared/test_sessions_metadata.py index a20650170..96533ac3d 100644 --- a/backend/tests/shared/test_sessions_metadata.py +++ b/backend/tests/shared/test_sessions_metadata.py @@ -140,6 +140,73 @@ async def test_user_stopped_upgrades_connection_lost(self, sessions_metadata_tab result = await get_session_metadata("i3", "u1") assert result.last_turn_interrupt_reason == "user_stopped" + @pytest.mark.asyncio + async def test_navigated_away_wins_over_connection_lost(self, sessions_metadata_table): + # The whole point of attesting departures: the cancellation backstop + # races the pagehide signal and must not erase it. Before the rank + # generalisation the condition only protected `user_stopped`, so this + # write order silently reverted to the unattributable reason. + from apis.shared.sessions.metadata import ( + store_session_metadata, + get_session_metadata, + set_interrupted_turn, + ) + await store_session_metadata(session_id="i5", user_id="u1", session_metadata=_make_session_metadata(session_id="i5")) + + await set_interrupted_turn("i5", "u1", reason="navigated_away", source="client_signal") + await set_interrupted_turn("i5", "u1", reason="connection_lost", source="cancellation") + + result = await get_session_metadata("i5", "u1") + assert result.last_turn_interrupt_reason == "navigated_away" + + @pytest.mark.asyncio + async def test_navigated_away_upgrades_connection_lost(self, sessions_metadata_table): + # Reverse order: the backstop lands first and the departure signal + # arrives late (the keepalive fetch outliving the page), upgrading it. + from apis.shared.sessions.metadata import ( + store_session_metadata, + get_session_metadata, + set_interrupted_turn, + ) + await store_session_metadata(session_id="i6", user_id="u1", session_metadata=_make_session_metadata(session_id="i6")) + + await set_interrupted_turn("i6", "u1", reason="connection_lost", source="cancellation") + await set_interrupted_turn("i6", "u1", reason="navigated_away", source="client_signal") + + result = await get_session_metadata("i6", "u1") + assert result.last_turn_interrupt_reason == "navigated_away" + + @pytest.mark.asyncio + async def test_navigated_away_never_downgrades_user_stopped(self, sessions_metadata_table): + # Stop, then the user closes the tab. The deliberate rejection is the + # stronger statement and must survive. + from apis.shared.sessions.metadata import ( + store_session_metadata, + get_session_metadata, + set_interrupted_turn, + ) + await store_session_metadata(session_id="i7", user_id="u1", session_metadata=_make_session_metadata(session_id="i7")) + + await set_interrupted_turn("i7", "u1", reason="user_stopped", source="client_signal") + await set_interrupted_turn("i7", "u1", reason="navigated_away", source="client_signal") + + result = await get_session_metadata("i7", "u1") + assert result.last_turn_interrupt_reason == "user_stopped" + + @pytest.mark.asyncio + async def test_unrecognised_reason_falls_back_to_unknown(self, sessions_metadata_table): + from apis.shared.sessions.metadata import ( + store_session_metadata, + get_session_metadata, + set_interrupted_turn, + ) + await store_session_metadata(session_id="i8", user_id="u1", session_metadata=_make_session_metadata(session_id="i8")) + + await set_interrupted_turn("i8", "u1", reason="something_new", source="cancellation") + + result = await get_session_metadata("i8", "u1") + assert result.last_turn_interrupt_reason == "unknown" + @pytest.mark.asyncio async def test_set_noop_when_session_missing(self, sessions_metadata_table): from apis.shared.sessions.metadata import set_interrupted_turn, get_session_metadata diff --git a/backend/uv.lock b/backend/uv.lock index 40be34623..7b769de8c 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -12,7 +12,7 @@ resolution-markers = [ [[package]] name = "agentcore-stack" -version = "1.14.1" +version = "1.15.0" source = { editable = "." } dependencies = [ { name = "aiofiles" }, diff --git a/docs/kaizen/research/2026-08-14.md b/docs/kaizen/research/2026-08-14.md new file mode 100644 index 000000000..af0d40baa --- /dev/null +++ b/docs/kaizen/research/2026-08-14.md @@ -0,0 +1,390 @@ +# Kaizen Research โ€” Friday, August 14, 2026 + +> Scan window: **July 24 โ€“ August 14, 2026 (21 days)**. Widened from the usual 7: the last research run was `2026-07-24`, so two Fridays were missed. Internal signal is reported over the full 21 days; external sources were scanned over 21 days with priority on the last 7. +> Web budget: **~71 / 50** used (over target โ€” justified by the 3ร— window; see Web Budget). + +## TL;DR + +The MCP **2026-07-28 spec went final and is now the Current version** โ€” it deletes the `initialize`/`initialized` handshake and `Mcp-Session-Id`, which means the `serverInfo` read our MCP Apps host uses to populate the App-frame header ([`mcp_apps.py:673`](backend/src/agents/main_agent/integrations/mcp_apps.py:673)) is a call against a protocol surface that no longer exists. That's the week's one confirmed breaking change against shipped code. But the highest-*upside* item is Anthropic's **mid-conversation tool changes without invalidating the prompt cache** (beta, shipped alongside Opus 5) โ€” a direct assault on the constraint that shapes this entire codebase's prompt-cache contract. Recommended #1 is a **cheap probe** of whether that beta is reachable through Bedrock Converse, because if it is, three separately-queued cost items collapse into one. + +Internally the news is good and the queue is stale: the `bedrock-agentcore` bump queued for **seven consecutive weeks shipped** (1.9.1 โ†’ 1.21.0, now at *zero* version lag), Strands went 1.48 โ†’ 1.51, and Nightly has been **green 12 days straight**. Several review-queue entries now assert things that are no longer true. + +## External Scan + +### What's moving this week + +Three weeks of accumulation, and the shape is unusually coherent: **the ecosystem spent this window attacking the prompt cache as a design constraint, from four independent directions.** Anthropic shipped mid-conversation tool mutation that doesn't bust the prefix, and a `cost_optimization` cookbook whose headline result is a 44% swing from moving a timestamp out of a system prompt. Claude Code shipped agent *staggering* so a fan-out's second agent reads the prefix the first one wrote, and fork-subagents that **inherit the cache** instead of rebuilding. MCP shipped cacheable list results with server-advertised TTLs. pydantic-ai shipped tools that stay hidden from `toolConfig` until searched for. Every one of those is a different answer to the same question we've been answering by hand for months โ€” and the convergence is itself the signal: our prompt-cache contract is not local paranoia, it is the industry's current hard problem. + +The second theme is **cost control moving out of the application layer**. AgentCore shipped customer-defined Gateway rate limits scoped by JWT claim with *token*-per-minute enforcement, plus temporal policies that evaluate a request against prior actions in the same session. pydantic-ai added `cost_limit` to `UsageLimits`. Claude Code surfaces gateway spend caps with reset times. Our quota runway โ€” shipped two weeks ago โ€” is entirely app-layer and advisory; the ecosystem is putting the same caps somewhere the model can't reason around. + +The surprise was the **MCP spec's asymmetry**: `server/discover` is *mandatory for servers to implement, optional for clients to call*. That's a deliberate escape hatch, and it means our migration is less urgent than "breaking change" implies โ€” but it also means a server can legally drop the old handshake tomorrow. + +### Notable items by source + +#### AWS Bedrock / AgentCore + +- **AgentCore Runtime "Instances" compute type GA** โ€” a second Runtime compute type running agents on AWS-managed EC2 *in your own account* via a capacity provider; persistent sessions up to 14 days, GPU types, multiple agents per instance, and โ€” critically โ€” **Savings Plans / ODCR coverage**. Bills as EC2 cost + a 12% management fee, ~1-minute minimum. โ€” https://aws.amazon.com/about-aws/whats-new/2026/08/aws-bedrock-agentcore-runtime-instances-generally-available/ โ€” *relevance*: the AgentCore Runtime construct in `infrastructure/lib/constructs/inference-api/`; this is the **first real lever on the W5 gap** (runtime memory โ‰ˆ 73% of the bill), because per-microVM-second billing with no commitment discount is exactly what Instances replaces. The 14-day persistent session also changes the calculus behind the idle-reaper work (#827). โ€” *unlocks*: reserved/discounted long-lived agent compute with in-account data residency, instead of pay-per-microVM-second. +- **Temporal policies + customer-configurable Gateway rate limiting** โ€” AgentCore Policy gains *stateful* authorization (evaluate a request against the agent's prior actions in the same session: enforce tool-call sequencing, require an argument to match a prior call's output, require human approval before privileged actions). Separately Gateway rate limits become customer-defined, scoped by JWT claim (`$.context.jwt.sub`), IAM principal, target, tool, or model ID โ€” with RPS/RPM, **TPM (token)**, and connection-rate enforcement, most-specific-match-wins, and `rate=0` as an emergency block. โ€” https://aws.amazon.com/about-aws/whats-new/2026/08/temporal-policies-agentcore/ ยท https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-rate-limits.html โ€” *relevance*: the quota runway (`QUOTA_RUNWAY_ENABLED`, `quota_session_notice`) and the drafted cooldown-windows spec are entirely application-layer today. Temporal policies also overlap the tool-approval / interrupt-resume path and per-tool MCP enablement (`toolId::name`). โ€” *unlocks*: per-user token budgets and stateful tool-sequencing guards enforced **outside agent code**, where the model cannot reason around them. +- **Runtime API rate quotas consolidated โ€” and new-session creation is now account-wide** โ€” data-plane calls including `InvokeAgentRuntime` share one adjustable **1,000 TPS/account** quota (up from 200). But new session creation becomes a single **25 TPS *per account*** quota shared across all endpoints, replacing the old per-endpoint 400 TPM / 25 TPS limits. Control-plane APIs collapse into three shared non-adjustable groups (mutations 50 TPS, Gets 150, Lists 25). โ€” https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/release-notes.html โ€” *relevance*: session creation is now an account-wide ceiling rather than per-endpoint headroom. Worth checking against the session single-flight guard and the tab-switch duplicate-invocation path, both of which create sessions. The 25 TPS List group is the one to watch for any admin page enumerating runtimes in a loop. +- **Runtime unified span destination โ€” default flipped July 20, 2026** โ€” Runtime spans now land in the agent's own log group (`/aws/bedrock-agentcore/runtimes/-`, `spans` stream) rather than the shared `aws/spans` group; **agents created on or after July 20 default to the agent's own group**, older agents keep the shared one unless `UNIFIED_TRACES_DESTINATION_ENABLED=true`. โ€” https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html โ€” *relevance*: this is the log group the `/ping` lifetime instrument and the AgentCore-424 traceback hunt both read. Any runtime recreated after July 20 silently changed where its spans land โ€” a telemetry query still pointing at `aws/spans` reads **empty rather than erroring**. Directly compounds the class of bug PR #843 just fixed (three dashboard widgets querying a log group nothing wrote to). +- **Bedrock IAM-principal cost allocation extended to `bedrock-mantle` endpoints** โ€” cost allocation by IAM principal now covers inference through the mantle endpoint, not just bedrock-runtime. โ€” https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-bedrock-expands-iam-principal-cost-allocation-bedrock-mantle/ โ€” *relevance*: `bedrock_mantle_config` and the admin cost surfaces. Gives a **billing-side cross-check** against our self-computed per-session cost rows โ€” useful given the known AdminUsageAggregates 2.6ร— triple-count. + +#### Strands Agents + +- **1.52.0 (Aug 12) โ€” ModelRouter + AgentStreamStage middleware with "middleware-initiated interrupts"** โ€” one minor ahead of our 1.51.0 pin; adds dynamic model selection via `Agent(model=)`, a stream-stage middleware layer that can interrupt a turn, Bedrock request-cancellation improvements, and logging for malformed tool-input JSON. โ€” https://github.com/strands-agents/sdk-python/releases โ€” *relevance*: middleware-initiated interrupts sit exactly where our interrupt-resume streaming (OAuth / tool-approval) and session-lease cancellation live; the Bedrock cancellation fix touches the same path as the dropped-SSE lease leak (#863) we shipped this week. โ€” *unlocks*: a supported interrupt primitive instead of hand-rolled anyio cancel-scope plumbing. +- **Issue #3758 (open, filed Aug 11) โ€” `cacheConfig` per-section TTLs can emit a checkpoint order Bedrock rejects on *every* request** โ€” setting `messagesTTL: '1h'` while tools are present produces tools@5m โ†’ messages@1h, violating Bedrock's non-increasing-TTL rule and returning `ValidationException` on the first and every subsequent turn. No construction-time validation; affects Python and TS. Workaround is `messagesTTL: false`. โ€” https://github.com/strands-agents/sdk-python/issues/3758 โ€” *relevance*: **a hard blocker on the layered-TTL technique the Anthropic cookbook recommends below.** Verified locally: we set `CacheConfig(strategy="auto")` with no per-section TTL ([`model_config.py:389`](backend/src/agents/main_agent/core/model_config.py:389)), so we are not exposed today โ€” but this is the landmine to walk around if the cookbook's mixed-TTL idea gets picked up. +- **Issue #3348 (our rolling-cachePoint request) โ€” still open, zero movement since July 20** โ€” no maintainer response, no comments, no linked PR. `CacheConfig(strategy="auto")` still strips prior message checkpoints, so a wide parallel tool fan-out (~20+ new blocks) pushes the previous checkpoint out of Anthropic's lookback and converts a read into a full re-write. โ€” https://github.com/strands-agents/sdk-python/issues/3348 โ€” *relevance*: confirms the three-cachePoint resilience workaround stays custom code. Nothing to delete; a bump-and-ping is the only cheap action. +- **1.51.0 โ€” the version we already run โ€” shipped three primitives we may be duplicating** โ€” selective offloading via a `should_offload` callback, MCP tool filtering with name prefixes, a snapshot session manager, plus `BeforeToolsEvent`/`AfterToolsEvent` batch hooks and `estimateUtilization()` on the Model base class. โ€” https://github.com/strands-agents/sdk-python/releases โ€” *relevance*: MCP name-prefix filtering overlaps our scoped-id `toolId::name` filtering at `_build_filtered_tools`; `should_offload` overlaps the queued `ContextOffloader` spike's selection logic. โ€” *unlocks*: possible deletion of custom filtering/offload-selection code. **This is an internal audit, not an upgrade** โ€” we already have these. +- **`strands-agents-tools` 0.8.6 (Aug 7) โ€” zero lag.** Our pin matches latest exactly. + +> โš ๏ธ **Sourcing caveat carried forward:** `sdk-python` now publishes **monorepo-wide** release notes (tags are `python/vX`, `typescript/vX`, `mcp/vX`) and `CHANGELOG.md` on `main` now **404s** โ€” the file is gone. Per this repo's own prior lesson (`project_strands_agentcore_upgrade_1_51` โ€” "diff the wheels, release notes are monorepo-wide"), diff 1.51.0 โ†’ 1.52.0 wheels before acting on any specific feature claim above. + +#### Reference repo (aws-samples/sample-strands-agent-with-agentcore) + +Very active โ€” ~30 commits, PRs #230โ€“#253, most recent `cd1a358` on 2026-08-12. + +- **`44944bf` feat(agent): route delegated work by task complexity (PR #247, Aug 11)** โ€” a shared model catalog carrying provider/family metadata plus explicit low/medium/high **delegation tiers**; both general delegation and the code-agent path classify a task, resolve a model from the tier, and **persist the resolution so retries reuse the same model**. โ€” https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/247 โ€” *applicability*: **highest-value item from this repo this window.** We pick the model from RBAC + user selection, never from task shape, and `@`-mention turns inherit whatever the parent had. Caveats before porting: their catalog is *a list that reaches the prompt* โ€” under our cache contract it must be deterministically ordered, and per-turn model swaps invalidate the prefix, so tiering must happen at **delegation boundaries**, not mid-session. The "persist the resolution for retry consistency" detail is worth copying verbatim โ€” same bug class as our version-in-cache-key trap. +- **`aed7d9c` fix(chat): harden session durability and job recovery (PR #249, Aug 11)** โ€” durable claim + **heartbeat + stale-takeover** + retry + cancellation + **fencing** on long-running jobs; conversation-**epoch** fencing on foreground memory writes. โ€” https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/249 โ€” *applicability*: directly parallel to our single-flight lease and the dropped-stream lease leak (#863). Two things they have that we don't: (1) a heartbeat with stale-takeover rather than a fixed-TTL lease โ€” strictly better than 90s expiry for "the holder died mid-stream"; (2) **epoch fencing on memory writes**, which is exactly the guard our one-session-two-cached-agents hazard (#741/#751) keeps needing hand-rolled per call site. Their stated limitation matches ours: foreground event buffers stay process-local, so mid-stream resume after process loss is still impossible. +- **`dd9c110` surface unseen session activity (PR #250) + `a616e84` prevent duplicate stream replay across sessions (PR #248, Aug 11)** โ€” unread/activity indicators for sessions progressing while the user is elsewhere, plus a fix for a stream replayed into the wrong session. โ€” https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/248 โ€” *applicability*: we deliberately keep concurrent streaming per-session and don't abort on navigate-away, which produces exactly the UX gap #250 fills โ€” a thread finishes in the background with no signal. Low-risk, user-visible. +- **`c3b2655` stream tool calls and arguments + `ff4a838` harden AG-UI protocol compatibility (PR #251)** โ€” https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/251 โ€” *applicability*: **we already have this and are arguably ahead** โ€” `ui_tool_input_partial` does partial-argument streaming with server-side JSON healing. The only delta is that they conform to **AG-UI** as a named wire protocol while we carry a bespoke SSE event set. Not worth porting; worth noting as a standards-convergence signal. +- **Workspace / Code-Interpreter cluster (PRs #232โ€“#234, #253, Aug 9โ€“12)** โ€” persists the Code Interpreter session workspace, adds a **storage-neutral session file API**, an S3-backed Files workspace, and a frontend workspace browser. โ€” https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/253 โ€” *applicability*: overlaps our session workspace tools spec (PR-1 built, gate key `workspace_files`). Their storage-neutral file API is worth comparing before we finish PR-2. **Flagging that three of their six workspace commits are security fixes** โ€” path normalization without regex backtracking, hashed storage keys, hardened file access. If our workspace tools expose user-supplied paths, those are the three holes to check. + +#### MCP ecosystem + +**Spec status โ€” 2026-07-28 is FINAL and Current.** + +| Question | Answer | +|---|---| +| Did 2026-07-28 go final? | **Yes.** Version string `2026-07-28`; it is the *Current* protocol version. | +| SEP-2575 (handshake + `Mcp-Session-Id` removal)? | **Landed.** Version/identity/capabilities now ride per-request `_meta` (`io.modelcontextprotocol/protocolVersion`) + the `MCP-Protocol-Version` header. `server/discover` is the replacement โ€” **mandatory for servers, optional for clients.** | +| SEP-1865 MCP Apps official? | **Yes, an official extension.** Apps spec dated 2026-01-26; SDK `@modelcontextprotocol/ext-apps` v1.1.2. **Capability identifier: could not confirm** โ€” do not code against `io.modelcontextprotocol/ui` *or* `experimental.ui` until verified against the apps spec source. | +| SEP-2567 (stateless) / SEP-2322 (multi-round-trip)? | **Both landed.** SEP-2567 ships jointly with 2575. SEP-2322 shipped as **Multi Round-Trip Requests (MRTR)**. | + +- **The `initialize` handshake is gone** โ€” https://modelcontextprotocol.io/specification/versioning โ€” *relevance*: **this breaks a documented assumption in shipped code.** [`mcp_apps.py:673`](backend/src/agents/main_agent/integrations/mcp_apps.py:673) captures `getattr(result, "serverInfo", None)` off `initialize` to populate the App-frame header's `serverName`/`icon`; that response no longer exists at 2026-07-28. `server/discover` returns capabilities and identity and is the migration target. Also collapses the "fresh MCP session per call" concern behind the MCP Apps proxy-call 504 work โ€” **there is no protocol-level session left to preserve.** +- **MRTR (SEP-2322) replaces held-open-stream serverโ†’client requests** โ€” server returns `resultType: "input_required"` with the requests it needs; client re-calls carrying `inputResponses`. โ€” https://blog.modelcontextprotocol.io/posts/2026-07-28/ โ€” *relevance*: the sanctioned shape for elicitation and, plausibly, for our `oauth_required` โ†’ resume interrupt and tool-approval pauses. โ€” *unlocks*: interrupt/resume **without holding an SSE stream open** against the 600s timeout โ€” directly relevant to the dropped-stream lease-leak class we just fixed. +- **Header-based routing: `Mcp-Method` + `Mcp-Name` now required on Streamable HTTP (SEP-2243)** โ€” gateways route and meter on headers instead of parsing JSON bodies. โ€” https://blog.modelcontextprotocol.io/posts/2026-07-28/ โ€” *relevance*: high, because we front MCP servers with AgentCore Gateway. โ€” *unlocks*: per-tool routing, metering, and authorization **at the Gateway edge** without body inspection โ€” a cleaner enforcement point than `toolId::name` filtering at `list_tools_sync`, and it composes with the new customer-defined Gateway rate limits above. +- **Cacheable list results (SEP-2549): `ttlMs` + `cacheScope` on tools/prompts/resources list responses** โ€” https://blog.modelcontextprotocol.io/posts/2026-07-28/ โ€” *relevance*: touches the prompt-cache contract directly. MCP tool listings reach `toolConfig`; a server-advertised TTL gives a principled refresh boundary instead of re-listing per turn. **The trap**: a TTL expiry that reorders or re-words the list is a full cacheable-prefix re-write at the $2.50/MTok write premium โ€” any adoption must keep deterministic ordering at the source. +- **MCP Apps host support is now broad and matrixed** โ€” Claude, Claude Desktop, VS Code Copilot, Microsoft 365 Copilot, Goose, Postman, MCPJam, Archestra.AI. โ€” https://modelcontextprotocol.io/extensions/apps/overview โ€” *relevance*: our host implementation is no longer bespoke; there is a published client matrix to conform to, plus an `AppBridge` module and `basic-host` example handling iframe sandboxing, message passing, tool-call proxying, and policy enforcement โ€” worth diffing against our hand-rolled proxy.html/`sandboxOrigin` path. +- **โš ๏ธ The documented `ui/` surface does not list `tool-input-partial`** โ€” the API overview enumerates `ui/initialize`, `ui/notifications/initialized`, `ui/notifications/tool-input`, `ui/notifications/tool-result`, `ui/notifications/resource-teardown`, `ui/message`, `ui/update-model-context`, `ui/open-link`. โ€” https://apps.extensions.modelcontextprotocol.io/api/documents/Overview.html โ€” *relevance*: we emit `ui_tool_input_partial` and relay `ui/notifications/tool-input-partial`. **This is not evidence it was removed** โ€” that page is an SDK overview, not the spec, and our implementation was built against SEP-1865 โ€” but the name's stability needs a direct check against the spec source. + +#### FastMCP + +- **FastMCP 4.0.0b1 โ€” sessionless transport for MCP 2026-07-28 (Jul 28, beta)** โ€” rebuilds the transport layer around the new protocol: no per-session state, no sticky sessions, requests route through any replica behind a load balancer, with per-connection negotiation so modern and handshake-era clients are both served. โ€” https://github.com/jlowin/fastmcp/releases/tag/v4.0.0b1 โ€” *implications*: the single most relevant FastMCP change of the window. Our Lambda-backed servers behind Gateway are exactly the sessionless cold-start shape v4 targets, and dual-era negotiation means a v4-upgraded server keeps serving Gateway's current handshake-era client. **Still beta โ€” track, don't adopt; no stable 4.x exists.** +- **4.0.0b1 breaking changes for server authors** โ€” server-initiated **sampling and roots are removed** (they require a persistent connection), 3.x deprecated module shims gone, object-mode decorators removed, error codes now spec-compliant. โ€” https://github.com/jlowin/fastmcp/releases/tag/v4.0.0b1 โ€” *implications*: any of our servers calling back to the client for sampling would break on a v4 bump. Worth a one-line audit of the external servers before anyone upgrades. +- **New v4 server-side primitives worth watching** โ€” `UserSession`/`SessionId` keyed to authenticated identity, multi-round-trip tools via guard mode (SEP-2322), a background-tasks extension (SEP-2663, `fastmcp-tasks`), a server extension API for capability negotiation (SEP-2133), plus auth additions: server-side identity assertion (SEP-990), `require_roles`, incremental auth step-up (SEP-2350), and **routable transport headers for gateways (SEP-2243)**. โ€” *implications*: SEP-2243 and `require_roles` are the two most directly useful to a Gateway-fronted fleet; **background tasks (SEP-2663) is a candidate answer to the MCP-Apps proxy-call timeout** we already hit. +- **3.4.x stable maintenance: 3.4.5 โ†’ 3.4.7, all OAuth/security fixes** โ€” 3.4.7 (Aug 10) corrects CIMD `private_key_jwt` assertion audience validation; 3.4.6 (Aug 5) adds trusted-proxy support for OAuth metadata/JWKS fetches with SSRF protection; 3.4.5 (Jul 27) fixes JWKS key filtering, Azure scope fallback, deep-object query serialization. โ€” https://github.com/jlowin/fastmcp/releases โ€” *implications*: non-breaking. Only bites servers doing their own OAuth/JWKS (our forward-auth and 3LO targets); Gateway-SigV4 targets unaffected. + +**Version facts**: latest stable **3.4.7** (2026-08-10); latest pre-release **4.0.0b2** (2026-08-07). Not pinned here โ€” tracked for external-server behavior. **No MCP Apps / `ui://` server-side support appeared anywhere in this window**, and no Lambda adapter changes. + +#### Agentic UI/UX patterns + +- **AI SDK MCP Apps host renderer** โ€” https://ai-sdk.dev/docs/ai-sdk-core/mcp-apps โ€” *what it is*: Vercel's first-party host for the same `ui://`-resource spec we ship. `experimental_MCPAppRenderer` reads the resource, mounts a sandboxed iframe, creates the bridge, pushes both tool-input and tool-result notifications, and proxies allowed app-originated `tools/call` back through host-supplied handlers; `@ai-sdk/mcp` **splits model-visible from app-visible tools**. โ€” *fit*: pattern-only. Our `ui_resource` / `ui_tool_input_partial` path already covers resource read + inlining + partial input. The genuinely new idea is **app-visible-only tools** โ€” tools the iframe may call but that are filtered out of the model's `toolConfig`. That is simultaneously a capability gain *and* a prompt-cache/token win. โ€” *where it'd land*: MCP App iframe surface + `_build_filtered_tools`. +- **Tool-approval state machine with signed approval requests** โ€” https://ai-sdk.dev/docs/agents/tool-approvals โ€” *what it is*: a tool gated on `'user-approval'` emits a `tool-approval-request` stream part with an `approvalId`; the client renders parts in `state: 'approval-requested'` and answers via `addToolApprovalResponse`, auto-resuming once all approvals settle. `experimental_toolApprovalSecret` HMAC-signs the request server-side, binding the signature to tool name, call ID, **and input arguments** โ€” so an approved call can't be replayed with mutated args. โ€” *fit*: pattern-only (Angular: a new SSE event beside `oauth_required`, resolved via `beginContinuationStreaming` and a `signal()`-backed pending-approvals map). **The arg-binding HMAC is worth stealing outright.** โ€” *where it'd land*: OAuth consent surface, generalized to per-tool-call approval. +- **assistant-ui โ€” composer draft recovery on failed send** (`@assistant-ui/react@0.15.14`, 2026-08-12) โ€” https://github.com/Yonom/assistant-ui/releases โ€” *what it is*: a new `MessageNotSentError` that gives the composer its draft back when a send never reached the backend, rather than losing typed text into an optimistic bubble that then fails. โ€” *fit*: **direct port**, small and self-contained (hold the composer `signal()` value until the stream's first `message_start`, restore on transport failure). โ€” *where it'd land*: chat composer. **Directly complements the 409-on-duplicate-turn single-flight guard we shipped** โ€” that is precisely a "send never reached the backend" case, and today it eats the draft. +- **assistant-ui โ€” stream reader cleanup + thread-switch scope binding** (batch of 2026-08-12) โ€” https://github.com/Yonom/assistant-ui/releases โ€” *what it is*: releasing stream readers after completion, cancellation, **and error** paths, plus corrected event-listener scope binding on thread switch. โ€” *fit*: pattern-only โ€” audit that our SSE subscription teardown is symmetric across done / abort / error. โ€” *where it'd land*: SSE streaming state โ€” same failure family as the dropped-stream lease leak (#863) and the tab-switch duplicate invocation. +- **NN/g โ€” "How to Decide When an AI Tool Is Worth Keeping" (PROVE framework)**, 2026-08-07 โ€” https://www.nngroup.com/articles/prove-framework/ โ€” *what it is*: a lightweight rubric testing one tool against one task to produce a defensible provisional keep/drop decision. โ€” *fit*: not UI code โ€” methodology. โ€” *where it'd land*: a usable rubric for `kaizen-review-prep`'s Ship/Decline/Defer column, and a framing for per-tool usage surfacing. + +#### Frontier model announcements + +- **Claude Opus 5 (July 24)** โ€” new Opus-tier model (`claude-opus-5`), positioned as Fable 5-level intelligence at half the price, aimed explicitly at long-running agents; **$5/MTok input, $25/MTok output โ€” the same as Opus 4.8**, with a Fast mode at ~2.5ร— speed for 2ร— base price. Announcement cites **configurable effort settings** to trade intelligence against token spend. โ€” https://www.anthropic.com/news/claude-opus-5 โ€” *relevance*: a drop-in Opus upgrade at flat cost is the rare model swap needing no quota re-forecast. โ€” *unlocks*: an effort-tier knob per model/skill (cheap effort for routing/utility turns, full effort for the answer turn). **Could not confirm Bedrock availability or model ID** โ€” the announcement lists Claude API / Claude.ai / Code / Cowork only. Verify against the Bedrock catalog before touching the model registry. +- **โญ Mid-conversation tool changes without invalidating the prompt cache (beta)** โ€” shipped alongside Opus 5: developers can change which tools Claude can use mid-conversation **without busting the cached prefix**. July 24. โ€” https://www.anthropic.com/news/claude-opus-5 โ€” *relevance*: **the single most consequential item in this scan for this repo.** Our entire prompt-cache contract exists because `toolConfig` sits in the cacheable prefix, forcing deterministic ordering at every source and making any tool-set mutation a 30kโ€“150k-token re-write at the $2.50/MTok write premium. โ€” *unlocks*: per-turn tool filtering that is currently too expensive to do โ€” cross-source tool search for MCP token bloat, per-tool MCP enablement changes mid-session, and `@`-mention agent switches that today re-write the whole prefix **on purpose**. **Needs a probe**: confirm the beta header, whether Bedrock Converse exposes it at all (it may be Claude-API-only), and measure against `toolConfigHash` on the `C#` rows. +- **OpenAI cut GPT-5.6 prices, replaced Priority Processing with Fast mode (July 30)** โ€” "GPT-5.6 Luna costs 80% less, while GPT-5.6 Terra costs 20% less"; Fast mode delivers up to 2.5ร— standard speed at 2ร— price and supersedes Priority Processing. โ€” https://developers.openai.com/api/docs/changelog โ€” *relevance*: an 80% cut on Luna changes the cost ranking of Mantle-path models against Nova Micro for cheap utility calls (title generation, classification, compaction summaries). **If any Mantle config requests Priority Processing, that service tier is gone.** +- **Fast mode extended past 272K tokens (Aug 5); Ultrafast in limited preview (Aug 13)** โ€” Fast mode gained long-context support beyond 272K on GPT-5.6 Sol/Terra/Luna; Ultrafast for Sol ("up to 14ร— faster than Standard") entered limited preview. โ€” https://developers.openai.com/api/docs/changelog โ€” *relevance*: long-context latency is the practical ceiling on attachment-heavy sessions (11% of sessions, 31% of prod spend). A 2.5ร— speedup there is a real latency lever โ€” but at 2ร— price it is the **opposite** of a cost lever, so it belongs behind a per-tier policy, never a global default. +- **OpenAI production-pinning guidance reaffirmed (Aug 6)** โ€” `chat-latest` now points at the newest model in ChatGPT for Plus/Pro, but **GPT-5.6 Sol remains the production API recommendation.** โ€” https://developers.openai.com/api/docs/changelog โ€” *relevance*: a floating alias in the model registry would silently re-point production traffic and re-write the cacheable prefix on whatever day OpenAI moves it โ€” exactly the invisible cost regression `systemPromptHash`/`toolConfigHash` was built to catch. **Audit the Mantle registry for any non-pinned identifier.** + +> **Could not confirm**: Bedrock availability for Opus 5; any Bedrock-side caching TTL or explicit-breakpoint change in this window (nothing found either way). **Gemini**: no model release, context-window, or API change confirmed in-window โ€” DeepMind's blog returned only Gemini Robotics ER 2. **No model deprecations or EOL dates** affecting models we run. Opus 4.8 explicitly remains available as an Opus 5 fallback. Search aggregators asserted OpenAI deprecations (reusable prompt objects, Evals platform, Agent Builder, Assistants API sunset) that did **not** appear in the dated changelog โ€” treat as unconfirmed. + +#### Agent harness patterns + +- **Claude Code 2.1.232 โ€” subagent forking on by default, and forks inherit the prompt cache** (2026-08-13) โ€” fork subagents now inherit the full conversation *and the prompt cache* rather than starting a fresh prefix. โ€” https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md โ€” *relevance*: this is the upstream answer to our `@`-mention history fork and the measured full-prefix re-write. Their fix is **inheritance at fork time, not a second cached `Agent`.** โ€” *unlocks*: a "fork the session, don't rebuild the agent" option for `@`-mention turns โ€” the mentioned agent adopts the existing message list + cache point instead of paying `agentSwitchUsd`. +- **Claude Code 2.1.229 โ€” agent staggering so subsequent agents read the cached prefix instead of re-paying** โ€” parallel fan-outs are deliberately staggered so agent N+1 hits a warm cache written by agent N. โ€” https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md โ€” *relevance*: we have no equivalent anywhere we fan out (scheduled runs, multi-agent turns); concurrent starts each pay a cache **write**. โ€” *unlocks*: a cheap scheduling tweak โ€” serialize the first call of a fan-out group by shared prefix hash, measurable directly on `cacheStatus` / `cacheWriteInputTokens`. +- **pydantic-ai v2.26.0 โ€” tools hidden until revealed (`load_capability`, `ToolReturn.tools`)** (2026-08-07) โ€” function tools can be declared but withheld from `toolConfig` until the model searches for them or a tool return injects them; same release added `Model.resolve_prompt_cache_retention()`. โ€” https://github.com/pydantic/pydantic-ai/releases โ€” *relevance*: exactly the problem in our tool-search / MCP-token-bloat strategy. **Note the tension**: revealing a tool mid-session *mutates* `toolConfig` and busts our exact-prefix cache โ€” which is precisely what the Anthropic beta above would fix. Ideas only. +- **pydantic-ai v2.23.0 / Claude Code 2.1.225 โ€” cost as a first-class runtime limit** (2026-08-04) โ€” pydantic-ai added `cost` to `RunUsage` and `cost_limit` to `UsageLimits`; Claude Code added gateway spend-limit support to usage warnings, naming the cap, reset time, and operator message. โ€” https://github.com/pydantic/pydantic-ai/releases ยท https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md โ€” *relevance*: both converge on what our quota runway does, and **both surface the cap and reset time** โ€” the piece our anchored-window spec still owes users. Note issues #860/#861 below ask for exactly this. โ€” *unlocks*: validation that a mid-run **hard** `cost_limit` (not just a warning) is the industry direction for the $30 platform backstop. +- **Claude Code 2.1.222โ€“2.1.228 โ€” a permission/sandbox hardening cluster around delegated agents** (2026-08-04โ†’08) โ€” fixes for PreToolUse auto-allow hooks bypassing tool restrictions **in background agent tasks**, Bash permission bypass via hidden argument spellings and invisible-Unicode padding, workflow scripts escaping the sandbox via dynamic `import()`, and synced skills shadowing local commands. โ€” https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md โ€” *relevance*: the through-line โ€” *a delegated or imported execution context inherited a permission it was never granted* โ€” is the same class as `isPublic`-was-listing-only (#852) and the RBAC half-write trap. โ€” *unlocks*: a targeted audit question: does a **scheduled run** or an `@`-mentioned agent evaluate `grantedTools`/`grantedSkills` against the *invoking user's* role, or against whatever the parent turn resolved? + +> **Note**: https://www.anthropic.com/engineering had **no posts in the window** โ€” most recent is 2026-04-23. Nothing from the blog this cycle. + +#### opencode (anomalyco/opencode) + +All in-window releases are patch-level (v1.18.9 โ†’ v1.18.18), no feature headlines. **Nothing mapped to the tooling lens** โ€” the only tool-adjacent items were MCP transport fixes (SSE reconnection-loop prevention in v1.18.11, legacy MCP SDK compatibility in v1.18.9). + +- **Session compaction keeps complete recent turns + clearer summaries (v1.18.17, 2026-08-12)** โ€” the summarization boundary now falls on whole-turn edges rather than mid-turn. โ€” https://github.com/anomalyco/opencode/releases/tag/v1.18.17 โ€” *lens*: context โ€” *relevance*: directly parallels our compaction byte-stability work. **Turn-aligned boundaries are exactly the property that keeps a restored prefix byte-stable** โ€” worth checking whether our truncation anchor is turn-aligned or block-aligned. The "for smaller models" angle is also a cost lever: a cheaper summarizer model, which we already use for titles (Nova Micro) but not for compaction. +- **Repeated compaction retains earlier tool-call history instead of dropping orphaned results (v1.18.15, 2026-08-07)** โ€” on a second/third pass, earlier tool calls are carried into the summary rather than orphaning tool results. โ€” https://github.com/anomalyco/opencode/releases/tag/v1.18.15 โ€” *lens*: context โ€” *relevance*: same failure class as `_repair_tool_pairing`. Our compaction death-spiral incident shows **repeated compaction is where our cost damage happened** โ€” a check that our second-pass compaction doesn't orphan tool results *and* doesn't re-write the prefix each pass is a cheap audit. +- **Capped automatic session retries with jitter; cache-write usage tracked separately (v1.18.17 / v1.18.14)** โ€” https://github.com/anomalyco/opencode/releases/tag/v1.18.14 โ€” *lens*: cost โ€” *relevance*: cache-write-vs-read accounting is precisely our `wastedUsd` / `cacheStatus` instrumentation. Independent convergence on "measure cache writes as their own line item" **validates that instrumentation**. The retry cap is a gap on our side: an uncapped retry on a 424/throttle re-writes the whole cacheable prefix each attempt. + +#### LibreChat + +Only one release falls in-window: **v0.8.8-rc1 (2026-08-14)** โ€” https://github.com/danny-avila/LibreChat/releases/tag/v0.8.8-rc1 + +- **Agent run steering + queued messages** โ€” interrupt or steer an in-progress agent run and queue the next message, plus agent-posed **multi-question forms** (up to four related questions in one form) that pause for input/approval then resume. โ€” *lens*: a/b โ€” *relevance*: same problem space as our interrupt-resume streaming; "queue the next message" is a UX affordance we don't have, and the batched multi-question form is a **cheaper elicitation pattern than N round-trips** (pattern-only, React/Node). +- **Agent Builder redesign โ€” unified tools marketplace, background tools, experimental "Agent Plugins"** โ€” plugins bundle deployment Skills, MCP servers, and opt-in command hooks as one installable unit. โ€” *lens*: b/c โ€” *relevance*: direct parallel to our Agent Designer / marketplace and per-tool MCP enablement. The notable choice is packaging skill + MCP server + hooks as **one installable unit** rather than binding primitives individually โ€” the opposite of our RBAC-granted-primitive model. Worth a look as a **distribution format, not an access model.** +- **Stateful Code Interpreter sessions + agent-managed memory with per-agent isolation** โ€” plus fullscreen artifact previews, Mermaid SVG/PNG export, multi-part response editing. โ€” *lens*: a/d โ€” *relevance*: "reusable conversation workspace" maps onto our session workspace tools PR-1. **Per-agent memory isolation is a design question we'll hit if Memory Spaces gets un-dark-stopped** โ€” a shared memory across agents is exactly the cross-instance staleness class that has bitten us twice. + +#### Pricing / quota + +- **GPT-5.6 Bedrock price cuts, effective 2026-07-30 โ€” the only confirmed in-window price move** โ€” on-demand Bedrock inference dropped **80% for Luna, 20% for Terra**, matching OpenAI first-party. The Bedrock pricing page now also lists Z AI GLM 5 / GLM 4.7 Flash, DeepSeek v3.2, and Qwen3. โ€” https://aws.amazon.com/about-aws/whats-new/2026/07/openai-gpt-terra-luna-pricing-bedrock/ โ€” *relevance*: worth a cost-per-quality bake-off for the cheap auxiliary calls we already run on small models (title generation on Nova Micro, compaction summaries, LLM-judge). **But** our prompt-cache contract and `cachePoint` handling are Anthropic/Converse-specific โ€” a swap on the *main agent path* is not a drop-in. +- **AgentCore pricing structure (standing, no in-window change found)** โ€” Runtime/Browser/Code Interpreter microVMs: **$0.0895 per vCPU-hour + $0.00945 per GB-hour** (per-second, 1s min). Runtime **Instances**: EC2 cost + 12% management fee (7.8% GPU G-series), ~1-min minimum, persistent up to 14 days. **Memory: $0.25/1,000 new events (short-term); $0.75/1,000 records/month for built-in long-term strategies vs $0.25/1,000 for override/self-managed; $0.50/1,000 retrievals.** Gateway: $0.005/1,000 API invocations, $0.025/1,000 Search API, **$0.02 per 100 tools indexed/month**. Identity: $0.010/1,000 token requests, free through Runtime or Gateway. โ€” https://aws.amazon.com/bedrock/agentcore/pricing/ โ€” *relevance*: **two hard levers on the 73%-of-bill memory line.** (1) GB-hour is ~10.6% of a vCPU-hour and is billed for the microVM's whole life โ€” the idle-reaper fix (#827) that cut lifetimes to 18โ€“50 min should already show; right-sizing memory allocation is the next multiplier. (2) **Verified internally this run**: [`memory-construct.ts:77`](infrastructure/lib/constructs/agentcore/memory-construct.ts:77) configures **all three built-in strategies** (`semanticMemoryStrategy`, `summaryMemoryStrategy`, `userPreferenceMemoryStrategy`) โ€” the **$0.75 tier, 3ร— the $0.25 override/self-managed rate.** Gateway's per-tool indexing fee also puts a real dollar figure on the MCP tool-bloat work. +- **"Control agent behaviors and cost beyond a single action"** โ€” AWS's own writeup of the cost-control capability release. โ€” https://aws.amazon.com/blogs/machine-learning/control-agent-behaviors-and-cost-beyond-a-single-action-new-capabilities-in-amazon-bedrock-agentcore/ โ€” *relevance*: Instances change the memory bill's shape from "pay per second of every session's microVM" to "pay for a long-lived instance" โ€” **potentially better for many concurrent short sessions sharing an agent, worse for spiky low-utilization.** Deserves a real model against actual session-concurrency data before anyone acts. +- **โš ๏ธ Could not confirm current Claude on-demand or prompt-caching prices** โ€” two extraction passes over https://aws.amazon.com/bedrock/pricing/ returned only Claude 3.5 Sonnet rows at figures inconsistent with known US rates; the page is JS-heavy with per-region tabs. **No cache write premium, read discount, or TTL term should be quoted from this scan.** The CLAUDE.md $2.50/MTok figure was *not* re-verified and no evidence of a change was found. โ€” *action*: re-verify via the **AWS Price List API** (`aws pricing get-products --service-code AmazonBedrock`) rather than scraping the marketing page โ€” that also produces a diffable artifact for future scans, strictly better than re-scraping weekly. + +#### Community + GitHub issues + +- **`bedrock-agentcore` #629 โ€” spans lost, TracerProvider never flushed before microVM freeze** (open, updated Aug 10) โ€” `_handle_invocation` returns without flushing OTel, so spans buffered at the end of an invocation are discarded when the microVM freezes. โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/629 โ€” *relevance*: our cost/cache observability leans on per-call telemetry; if the tail of each invocation is silently dropped, EMF/trace-derived numbers **under-report exactly the last turn of a session** โ€” which is where compaction and cache-write spikes land. +- **`bedrock-agentcore` #621 โ€” `filter_restored_tool_context` incompatible with extended thinking** (open, updated Aug 8) โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/621 โ€” *relevance*: direct hit on the session-restore path `TurnBasedSessionManager` depends on. Any mutation of restored history is a **prompt-cache prefix rewrite before it is a correctness bug.** +- **`bedrock-agentcore` #564 โ€” history silently dropped on inconsistent `ListEvents`** โ€” still **open**; the only one of our three tracked issues without a fix. Metadata-filtered `ListEvents` transiently misses marker events, so the manager treats the turn as a new session and skips history restoration though the data exists unfiltered. โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/564 โ€” *relevance*: a false "new session" both loses context and re-writes the entire cacheable prefix. Worth an explicit guard on our side rather than waiting on upstream. +- **`bedrock-agentcore` #583 โ€” `StrandsA2AExecutor` via `serve_a2a` never hits the AgentCore idle session timeout** (open, updated Jul 27) โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/583 โ€” *relevance*: pairs with our CLAUDE.md note that A2A is client-only today. **If/when we expose an A2A server, this reintroduces the runaway-microVM class we just fixed with the idle reaper (#827)** โ€” alongside the existing `streaming=True` requirement already documented there. +- **starter-toolkit #498 โ€” no API to list or force-terminate active runtime sessions** (open, updated Jul 12) โ€” https://github.com/aws/bedrock-agentcore-starter-toolkit/issues/498 โ€” *relevance*: confirms there is still **no upstream operator kill path** for a runaway session; our quota/cooldown and lease work remains the only backstop. +- **HN: "qm โ€” multiplayer agent harness for work"** (682 pts, 164 comments, 2026-07-31) โ€” https://news.ycombinator.com/item?id=49126604 (repo: https://github.com/yc-software/qm) โ€” *relevance*: worth a real read next scan. "Multiplayer" is exactly our one-session-many-agents problem; any harness solving concurrent actors on one session state is a direct comparison for `TurnBasedSessionManager`. *(Metrics and title only โ€” the thread was not read.)* +- **HN: Docker Sandboxes (692 pts, 2026-08-10) and Cloudflare OS (664 pts, 2026-08-05)** โ€” https://news.ycombinator.com/item?id=49239751 ยท https://news.ycombinator.com/item?id=49182996 โ€” *relevance*: both occupy the slot AgentCore Code Interpreter / Browser and our mcp-sandbox stack occupy. Credible non-AWS options are cost/lock-in datapoints for the sandbox line item. +- **Notably: AWS's own "Runtime instances" HN post drew 1 point, 0 comments** โ€” https://news.ycombinator.com/item?id=49207973 โ€” *relevance*: **zero community validation.** Treat the Instances claims as vendor-only until someone reports real numbers โ€” which is an argument for modelling it ourselves rather than trusting the pitch. + +#### Cookbook / courses + +- **โญ `cost_optimization/cost_optimization.ipynb` โ€” new cookbook, Aug 12 2026** โ€” seven measured cost strategies benchmarked against a 10-claim eval baseline on Opus, with a `usage_cost()` helper converting cache-write/cache-read/TTL-multiplied token counts into dollars. โ€” https://github.com/anthropics/claude-cookbooks/blob/main/cost_optimization/cost_optimization.ipynb โ€” *relevance*: **the most directly applicable artifact to our prompt-cache contract that has ever appeared in this scan.** Four techniques map one-to-one: + 1. **Byte-stable prefixes** โ€” demonstrates a **44% swing purely from moving `datetime.now()` out of the system prompt** into the user message. Exactly the bug class `systemPromptHash` exists to catch. + 2. **Layered breakpoints with mixed TTLs** โ€” a 1-hour-TTL point on the static policy manual behind an ephemeral point on per-request context: **54% cheaper** than a single flat point. Evaluate against our 3-cachePoint resilience work โ€” **but see Strands #3758 above, which makes naive per-section TTLs a hard `ValidationException`.** + 3. **Progressive context disclosure** โ€” replacing an 11K-token inlined manual with a `read_manual` tool; the same shape as our `ContextOffloader` / attachment-cost problem. + 4. **Deferring non-core tools behind `tool_search` so the core `toolConfig` stays cached** โ€” the upstream-blessed version of our cross-source tool-search strategy. + It **explicitly declines model-downgrading and the Batch API as quality-eroding** โ€” consistent with our "quality wins when cost and quality genuinely conflict" tenet. +- **Dynamic workflows cookbook, Agent SDK series (PR #806, Aug 3)** โ€” https://github.com/anthropics/claude-cookbooks/commit/3291e01531b44bcf730ffe5ed62df60088087316 โ€” *relevance*: agent-loop composition reference. The notebook itself was not opened โ€” treat technique detail as unverified. +- **Managed Agents cookbooks: budgets, advisor, repo skills, inference geo (Aug 6)** โ€” https://github.com/anthropics/claude-cookbooks/commit/215d95722a18105cdcee2026f840eeae93f1330d โ€” *relevance*: the **budgets** notebook is the closest upstream analogue to our per-user quota + $30 platform-ceiling spec; worth comparing enforcement points (pre-call budget check vs. post-hoc aggregation). Managed Agents is a hosted-sandbox product we don't use, so the surrounding harness code isn't portable. + +#### Seasonal + +Out of window โ€” no seasonal sources scanned this week. (re:Invent is late Nov / early Dec; no conference proceedings due in August.) + +### Patterns worth considering + +- **"The prompt cache is the constraint" is now an industry-wide design axis, not our local quirk.** Four independent vendors shipped four different answers in three weeks: Anthropic (mutate tools without invalidating), Claude Code (stagger fan-outs; forks inherit the cache), MCP (server-advertised list TTLs), pydantic-ai (tools hidden until revealed). + - **Where**: https://www.anthropic.com/news/claude-opus-5 ยท https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md ยท https://blog.modelcontextprotocol.io/posts/2026-07-28/ ยท https://github.com/pydantic/pydantic-ai/releases + - **Fit**: strongly validating. Our CLAUDE.md prompt-cache contract, `cacheStatus`/fingerprint observability, and `partial_miss` classification are ahead of most of these โ€” we can *measure* what they are guessing at. What we lack is the **mutation escape hatch**: everyone else is buying the ability to change the tool set cheaply, and we've been paying full price to avoid changing it at all. + - **Verdict**: **Worth trying** โ€” via the probe in Idea #1. If the Anthropic beta is reachable on Bedrock, three separately-queued items (cross-source tool search, per-tool MCP enablement, `@`-mention prefix cost) stop being blocked by the same wall. + +- **Cost enforcement is migrating below the application layer.** AgentCore Gateway rate limits with per-JWT-claim **TPM** caps and `rate=0` emergency block; AgentCore temporal policies; pydantic-ai `cost_limit`; Claude Code gateway spend limits that name the cap *and the reset time*. + - **Where**: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-rate-limits.html ยท https://github.com/pydantic/pydantic-ai/releases + - **Fit**: our quota runway is advisory and app-layer โ€” a compromised or looping agent path routes around it. A Gateway-level TPM cap keyed on `$.context.jwt.sub` is enforced where agent code cannot reach. Note this arrives the same week two users independently asked for **live spend visibility** (#860, #861) โ€” the surfacing half and the enforcement half both have fresh signal. + - **Verdict**: **Monitor โ†’ scope next week.** Real, but it lands on top of a drafted-not-shipped cooldown spec; sequencing matters more than speed. + +- **Turn-aligned compaction boundaries as a byte-stability property.** opencode moved its summarization boundary to whole-turn edges; their next release fixed repeated compaction orphaning earlier tool results. + - **Where**: https://github.com/anomalyco/opencode/releases/tag/v1.18.17 + - **Fit**: cheap and directly checkable โ€” is our truncation anchor turn-aligned or block-aligned, and does a *second* compaction pass preserve tool pairing without re-writing the prefix? Our own death-spiral incident says repeated compaction is where the damage happens. + - **Verdict**: **Worth trying** as an audit (a few hours), folded into Idea #4. + +- **App-visible-only tools** (Vercel `@ai-sdk/mcp`): tools the MCP App iframe may call that are filtered *out* of the model's `toolConfig`. + - **Where**: https://ai-sdk.dev/docs/ai-sdk-core/mcp-apps + - **Fit**: a capability gain and a token/cache win simultaneously โ€” an App can expose rich interaction without every one of its tools bloating the cacheable prefix. Lands on `_build_filtered_tools`, which already does scoped-id filtering. + - **Verdict**: **Worth trying** โ€” but sequence behind the MCP host migration (Idea #2), same file neighbourhood. + +- **Composer draft recovery on failed send** (assistant-ui `MessageNotSentError`). + - **Where**: https://github.com/Yonom/assistant-ui/releases + - **Fit**: **direct port, genuinely small.** Hold the composer `signal()` until the first `message_start`; restore on transport failure. This is the missing half of the 409-duplicate-turn guard we shipped in 1.14.1 โ€” we now correctly *reject* the duplicate and still lose the user's typed text. + - **Verdict**: **Worth trying** โ€” best effort-to-user-visible-value ratio in the whole scan. Not in the Top 5 only because it competes against structurally larger items; it should be picked up regardless. + +## Internal Audit + +### Activity (July 24 โ€“ August 14, 21 days) + +- **Commits on develop**: 151 (non-merge) +- **PRs merged into develop**: 43 โ€” **open now**: 2 ([#869](https://github.com/Boise-State-Development/agentcore-public-stack/pull/869) G3 citations probe, [#851](https://github.com/Boise-State-Development/agentcore-public-stack/pull/851) TUI client) +- **Releases cut**: 1.12.0, 1.12.1, 1.12.2, 1.12.3, 1.13.0, 1.14.0, 1.14.1 +- **Issues opened**: 8 โ€” **closed**: 12 โ€” **open total**: 49 +- **Reverts**: 1 (`405d3526` revert(auth): remove the public Cognito client for CLI PKCE auth โ€” a deliberate scope walk-back on #848, not a failure) +- **CI failures**: **zero in the last 12 days.** Nightly Build & Test has been green Aug 3 โ†’ Aug 14 (12/12). Every workflow on the last 20 runs is `success`. + +### Repeated friction signals + +- **Users are asking for live spend visibility โ€” twice, independently, in three days** ([#860](https://github.com/Boise-State-Development/agentcore-public-stack/issues/860) "real-time usage counter against monthly allowance on-screen", [#861](https://github.com/Boise-State-Development/agentcore-public-stack/issues/861) "automate end-user initiated quota increase requests", both 2026-08-12). + - **Hypothesis**: 1.14.0 shipped the quota *runway* โ€” 50%/75% rungs plus `quota_session_notice` โ€” which are **event-driven and dismissible**. Users evidently want an **ambient, always-visible** number, and a self-service path when they hit it. The warnings told them a wall exists without letting them see how close it is or do anything about it. + - **Fix candidate**: a persistent usage indicator reading the same `totalCost` aggregate `quota_session_notice` already uses (no new aggregation), plus a request-increase action on the quota surface. Note both pydantic-ai and Claude Code surface **the cap and its reset time** in-message โ€” our anchored-window spec owes exactly that. +- **Telemetry pointed at log groups nothing writes to โ€” twice, and upstream just moved the goalposts again.** PR #843 fixed three dashboard widgets querying a phantom log group (#697's `cacheStatus` widget had *never* shown data; #838 copied the pattern). This scan found AWS flipped the Runtime span destination default on **July 20** โ€” agents created after that date write spans to their own group. + - **Hypothesis**: empty query results read as "no errors / no traffic" rather than as a broken query. This is a **silent-failure pattern**, not a one-off bug โ€” it has now recurred with a different root cause. + - **Fix candidate**: any observability widget or Logs Insights query should assert *non-empty* on a known-good window in CI, or carry a visible "0 results โ€” verify source" affordance. Cheap; prevents a third instance. +- **Delegated/derived execution contexts inheriting permissions they were never granted.** Three of our own recent fixes are the same shape (#852 `isPublic` listing-only, the RBAC half-write trap, the `@`-mention agent fork), and Claude Code shipped a five-version cluster of exactly this class upstream. + - **Hypothesis**: our access checks are correct at the *primary* turn and under-specified at every derived entry point. + - **Fix candidate**: a targeted audit answering one question โ€” does a **scheduled run** or an `@`-mentioned agent evaluate `grantedTools`/`grantedSkills` against the *invoking user's* role, or against whatever the parent turn resolved? + +### Version-pin lag + +| Dep | Pinned | Latest | Lag | Notes | +|---|---|---|---|---| +| `bedrock-agentcore` | **1.21.0** | **1.21.0** | **none** โœ… | **The seven-week-queued bump shipped** (#857, 1.9.1 โ†’ 1.21.0). #482 and #571 both closed upstream. #564 remains open. | +| `strands-agents` | 1.51.0 | 1.52.0 | 1 minor / 5 days | 1.52.0 (Aug 12): ModelRouter, stream-stage middleware with interrupts, Bedrock cancellation fixes. Low-risk, lands on paths we just touched. | +| `strands-agents-tools` | 0.8.6 | 0.8.6 | **none** โœ… | Current. | +| `boto3` | 1.43.68 | 1.43.71 | 3 patches / 1 day | Trivial; near-daily API-model updates. Remember the floor is pinned in 4 files. | +| `fastapi` | 0.136.1 | 0.141.1 | 5 minors | Pre-1.0 semver โ€” each 0.x minor can carry breaking changes. Review notes; don't assume patch-safety. | +| `aws-cdk-lib` | 2.262.0 | 2.265.0 | 3 minors | Routine, non-breaking under CDK v2's stability contract. Bundles its deps โ€” overrides can't patch inside it. | +| `aws-cdk` (CLI) | 2.1128.0 | 2.1136.0 | 8 minors | CLI moves independently of the library; normally non-breaking. | +| `constructs` | 10.6.0 | 10.8.1 | 2 minors + patch | Non-breaking. Stay on 10.x โ€” `latest-3` is the legacy CDKv1 line. | +| `vitest` | 4.1.5 | 4.1.10 | 5 patches | Safe. **Signal**: `5.0.0-rc.1` is on the `rc` tag โ€” a v5 major is imminent, worth watching given our Vitest/Analog flake history. | +| `@angular/core` | 21.2.17 | **22.1.2** (latest) ยท 21.2.20 (v21-lts) | **1 major** | Angular 22.0.0 landed 2026-06-03. **v21 has moved to the `v21-lts` tag** โ€” i.e. into maintenance. Low-risk move is 21.2.17 โ†’ 21.2.20. | +| `typescript` | 5.9.3 | **7.0.2** | **2 majors / ~10.5 months** | **Biggest gap in the table.** 5.9.3 (2025-09-30) โ†’ 6.0.2 โ†’ 7.0.2 (2026-07-08). TS 7 is the native/Go compiler rewrite; TS 6 was the deprecation-carrying transitional major. Both explicit breaking majors โ€” **must be checked against Angular's supported-TypeScript range first**, which makes it downstream of the Angular decision. | + +> **Verification note**: every date in this table was re-verified against raw PyPI/npm registry JSON (`upload_time_iso_8601`, packument `time` map). A first-pass summarizer returned a **fabricated** upload date for fastapi; it was caught and corrected. Treat summarized package pages as unreliable for dates. + +### Retirement candidates + +- **โš ๏ธ The review queue itself is the top retirement candidate.** Its highest-priority entry โ€” repeated across four dated items ([2026-07-10], [2026-07-17], [2026-07-24]) โ€” asserts *"We're on 1.9.1, exposed to #482 + #571 today."* **That is now false**: #857 shipped 1.21.0, both issues are closed upstream, and lag is zero. Two more entries are similarly stale: the [2026-07-24] nightly `DELETE_FAILED` item (Nightly has been green 12 consecutive days) and the [2026-07-10] Strands-bump entry whose own status line already says "the bump itself SHIPPED". Four superseded entries chain through the same subject. **`kaizen-review-prep` runs against this queue in ~2 hours and will rank stale premises as live work unless they're resolved first.** +- **`.claude/skills/angualar-best-practices/`** โ€” unmodified since **2025-12-30** (7.5 months). Also note the directory name is misspelled (`angualar`), which means any doc or prompt referencing `angular-best-practices` silently misses it. Either refresh it (Angular 21 โ†’ 22 is now a live question) or retire it โ€” a stale Angular skill is worse than none while the framework is a major behind. +- **`.claude/skills/frontend-design/`** โ€” unmodified since **2026-01-18** (7 months), and duplicated by the `anthropic-skills:frontend-design` plugin skill now available in-session. Candidate for retirement in favour of the plugin version. +- **The curl version pin** ([`Dockerfile.app-api:42`](backend/Dockerfile.app-api:42) / [`Dockerfile.inference-api:42`](backend/Dockerfile.inference-api:42), still `curl=8.14.1-2+deb13u*`) โ€” queued [2026-07-17] and still open. The `deb13u*` wildcard is holding (Backend Deploy green all window), so this **drops in urgency** but not in correctness: the pin was never a supply-chain control, it's a HEALTHCHECK probe, and the wildcard only defers the break to the next Debian series bump. + +### Risks introduced this week + +- **MCP 2026-07-28 deleted the `initialize` handshake our MCP Apps host reads.** โ€” https://modelcontextprotocol.io/specification/versioning โ€” *what breaks if we ignore this*: any MCP server that upgrades to the new protocol stops returning `serverInfo`, so App frames silently lose their `serverName` and `icon` and fall back to the title-cased `ui://` authority + generic glyph. **Degrades quietly, doesn't error.** Mitigating factor: `server/discover` is optional for *clients*, so servers must keep serving handshake-era clients during the transition โ€” this is urgent-ish, not on fire. +- **`bedrock-agentcore` #629 โ€” end-of-invocation spans are dropped when the microVM freezes.** โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/629 โ€” *what breaks if we ignore this*: our cost and cache telemetry **under-reports the final turn of every session** โ€” precisely where compaction and cache-write spikes land. Any conclusion drawn from EMF/trace data about end-of-session cost may be systematically low. +- **AWS flipped the Runtime span destination default on 2026-07-20.** โ€” https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html โ€” *what breaks if we ignore this*: a runtime recreated after that date writes spans somewhere our queries may not look, and the failure mode is an **empty result, not an error** โ€” the exact pattern PR #843 just fixed for a different root cause. +- **`bedrock-agentcore` #564 remains open** (false "new session" on inconsistent `ListEvents`). โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/564 โ€” *what breaks if we ignore this*: silent context loss **and** a full cacheable-prefix re-write. This was the one of our three tracked issues the 1.21.0 bump did *not* close โ€” the bump's value is real but incomplete. +- **Uncommitted infrastructure changes are live in the working tree.** Three files (`app-api-iam-grants.ts` โ€” `s3vectors:DeleteVectors` for document cleanup; `inference-api-iam-roles.ts` โ€” `dynamodb:GetItem` on the user-settings table; `security-policy.test.ts`) carry real, commented IAM fixes that are **not committed to any branch**. Both fix silent-failure bugs (orphaned vectors surviving until TTL; a user's chosen default model ignored with only a stray ERROR line). *What breaks if we ignore this*: they're one `git checkout` from being lost. Left untouched by this run โ€” flagged for a home. + +## Ideas โ€” Top 5 (ranked) + +| # | Idea | Surface | Effort | Impact | Subtracts? | Unlocks? | +|---|---|---|---|---|---|---| +| 1 | Probe Anthropic's mid-conversation tool-mutation beta on Bedrock | backend | **L** (probe) | **H** | Conditionally โ€” could retire the "never mutate `toolConfig`" prohibition and the workarounds built around it | Per-turn tool filtering; cheap `@`-mention switches; MCP tool-bloat reduction | +| 2 | Migrate the MCP Apps host off `initialize`/`serverInfo` to `server/discover` | backend | M | Mโ€“H | Yes โ€” retires the `initialize`-serverInfo dependency and the unconfirmed `experimental.ui` id | Spec-final MCP conformance; MRTR elicitation; stateless-transport readiness | +| 3 | Attack the W5 memory bill: self-managed LTM strategies + model Runtime Instances | infra / backend | M | **H** | Yes โ€” a 3ร— per-record price cut on a line we're overpaying by default | Savings-Plans-covered agent compute; 14-day persistent sessions | +| 4 | Run the Anthropic `cost_optimization` cookbook as an audit against our own contract | backend | **L** | Mโ€“H | Yes โ€” finds and deletes cache-busting waste rather than adding anything | A measured, upstream-blessed baseline for the cost-effectiveness arc | +| 5 | Resolve the stale review queue before `kaizen-review-prep` consumes it | docs / process | **L** | Mโ€“H | Yes โ€” retires 4 superseded entries whose premises are now factually false | Trustworthy input to the ranking run 2 hours from now | + +--- + +### 1. Probe Anthropic's mid-conversation tool-mutation beta on Bedrock + +- **Source**: https://www.anthropic.com/news/claude-opus-5 (2026-07-24) โ€” "change which tools Claude can use mid-conversation without invalidating the prompt cache" (beta). Corroborated by convergent upstream work: pydantic-ai `ToolReturn.tools`/`load_capability` (v2.26.0), MCP SEP-2549 cacheable list results, Claude Code fork-inherits-cache (2.1.232). +- **Surface area**: `agents/main_agent/core/model_config.py` (Bedrock request construction), `_build_filtered_tools`, the `toolConfigHash` fingerprint on `C#` rows, `GET /admin/costs/sessions/{id}/calls`. +- **Change**: **a probe, not a build.** Answer three questions and write them down: (a) does Bedrock's Converse API expose the beta at all, or is it Claude-API-only? (b) if exposed, what is the header/parameter and does `strands` pass it through? (c) measured on a real session, does adding a tool mid-conversation leave `cacheReadInputTokens` intact? This repo has a strong track record here โ€” G1 disproved the agent-cache cost thesis and G3 disproved the citations premise. **A negative result is a valid, valuable outcome**: it closes the question for a quarter. +- **Subtracts**: conditionally, and enormously. Our prompt-cache contract's most expensive clause is "any list that reaches `toolConfig` must be deterministically ordered at its source" โ€” a constraint propagated across skills, tools, models, and MCP listings. If mutation is cheap, that clause narrows dramatically, and at least three queued items (cross-source tool search for MCP token bloat, per-tool MCP enablement, `@`-mention prefix cost) stop being blocked by the same wall. +- **Unlocks**: + - Per-turn tool filtering โ€” currently too expensive to contemplate โ€” which is the actual fix for MCP tool-list bloat. + - `@`-mention agent switches that don't deliberately re-write a 30kโ€“150k-token prefix (`agentSwitchUsd` becomes optional rather than structural). + - The Vercel "app-visible-only tools" pattern and pydantic-ai's tool-search pattern both become affordable. +- **Effort ร— Impact**: **Low ร— High.** A day of investigation against a capability that, if real on Bedrock, changes the cost shape of the platform. +- **Risk**: the most likely outcome is "Claude API only, not on Bedrock." That is still worth knowing definitively โ€” it's currently an open assumption sitting under several queued items. +- **Verdict**: **Worth trying โ€” recommended #1.** + +### 2. Migrate the MCP Apps host off `initialize`/`serverInfo` to `server/discover` + +- **Source**: https://modelcontextprotocol.io/specification/versioning ยท https://blog.modelcontextprotocol.io/posts/2026-07-28/ โ€” MCP 2026-07-28 is now the **Current** protocol version; SEP-2575 removed the `initialize`/`initialized` handshake and `Mcp-Session-Id`. +- **Surface area**: [`agents/main_agent/integrations/mcp_apps.py:673`](backend/src/agents/main_agent/integrations/mcp_apps.py:673) (the `getattr(result, "serverInfo", None)` capture), the `_mcp_apps_server_info` field and its consumers around lines 454โ€“461 and 628โ€“635, `streaming/stream_coordinator.py:1680` (`ui_resource` header emission), and the `ClientCapabilities(experimental=...)` subclassing noted at `mcp_apps.py:26`. +- **Change**: resolve `serverName`/`icon` from `server/discover` with the existing `manifest.json` fallback retained as the second tier and the title-cased `ui://` authority as the third. Keep the handshake path as a compatibility branch โ€” the spec makes `server/discover` **mandatory for servers but optional for clients**, so both eras must be served during the transition. **Do not change the capability identifier yet**: this scan could *not* confirm whether it is `io.modelcontextprotocol/ui` or still `experimental.ui`, and coding against the wrong one is worse than waiting. +- **Subtracts**: yes โ€” retires the `initialize`-response dependency, and collapses the "fresh MCP session per call" concern behind the MCP Apps proxy-call 504 work, since **there is no protocol-level session left to preserve**. +- **Unlocks**: conformance with a published host matrix (Claude, VS Code Copilot, M365 Copilot, Goose, Postman); readiness for **MRTR (SEP-2322)**, which is the sanctioned interrupt/resume shape and would let OAuth consent and tool approvals resume **without holding an SSE stream open** against the 600s timeout; and readiness for SEP-2243 header-based routing at the Gateway edge. +- **Effort ร— Impact**: Medium ร— Mediumโ€“High. +- **Prerequisite**: two verification items first, both cheap โ€” (a) confirm the UI capability identifier against the apps **spec source**, not the docs site; (b) confirm `ui/notifications/tool-input-partial` still exists in the spec (it is absent from the SDK overview page, which is *not* evidence of removal, but our `ui_tool_input_partial` relay depends on it). +- **Verdict**: **Worth trying** โ€” but do the two verifications before writing code. This absorbs and supersedes the [2026-07-24] "prep the MCP Apps host for the 2026-07-28 spec" queue item, which was written when the spec was still an RC. + +### 3. Attack the W5 memory bill: self-managed LTM strategies + model Runtime Instances + +- **Source**: https://aws.amazon.com/bedrock/agentcore/pricing/ (built-in long-term strategies **$0.75/1,000 records/month** vs override/self-managed **$0.25/1,000**) + https://aws.amazon.com/about-aws/whats-new/2026/08/aws-bedrock-agentcore-runtime-instances-generally-available/ (Instances GA, EC2 + 12% fee, Savings Plans / ODCR eligible, 14-day sessions). **Verified internally this run**: [`memory-construct.ts:77`](infrastructure/lib/constructs/agentcore/memory-construct.ts:77) configures all three built-in strategies. +- **Surface area**: `infrastructure/lib/constructs/agentcore/memory-construct.ts` (the `memoryStrategies` array and `eventExpiryDuration: 90`), `infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts` (Runtime compute type + memory allocation), `apis/app_api/memory/routes.py` (the `/facts/`, `/preferences/`, `/summaries/` namespace readers). +- **Change**: two independent tracks, cheapest first. + 1. **LTM strategy tier** โ€” we run `semanticMemoryStrategy`, `summaryMemoryStrategy`, and `userPreferenceMemoryStrategy`, all built-in, all at the $0.75 tier. Determine actual record volume per strategy and whether each is *read* often enough to justify 3ร— the self-managed rate. This is a straight price cut on records we're already storing, if the extraction quality holds. + 2. **Runtime Instances** โ€” model per-microVM-second billing against EC2 + 12% with Savings Plans coverage, using real dev/prod session-concurrency data. Instances favour **many concurrent short sessions sharing an agent** and penalize **spiky low-utilization**; which we are is an empirical question, not a guess. Also right-size the memory allocation โ€” GB-hour is ~10.6% of a vCPU-hour but bills for the microVM's whole life. +- **Subtracts**: yes on track 1 โ€” a 3ร— per-record price reduction on a line item we're paying the premium tier for by default, with no deliberate decision behind it. +- **Unlocks**: Savings-Plans/ODCR-covered agent compute (the first commitment-discount lever AgentCore has ever offered us) and 14-day persistent sessions, which changes the calculus behind the idle-reaper work. +- **Effort ร— Impact**: Medium ร— **High** โ€” this is the named W5 gap in the cost-effectiveness roadmap, and it is the first week the ecosystem handed us an actual lever on it. +- **Caution**: **AWS's own Instances announcement drew 1 point and 0 comments on HN** โ€” there is zero independent validation of the cost claims. Model it against our own numbers; do not adopt on the pitch. Track 1 (the $0.75โ†’$0.25 question) is the safer, faster half and should go first. +- **Verdict**: **Worth trying** โ€” split the tracks; ship track 1's *measurement* first. + +### 4. Run the Anthropic `cost_optimization` cookbook as an audit against our own contract + +- **Source**: https://github.com/anthropics/claude-cookbooks/blob/main/cost_optimization/cost_optimization.ipynb (2026-08-12) โ€” seven measured strategies with a `usage_cost()` helper. Reinforced by opencode's turn-aligned compaction (v1.18.17) and retry-cap-with-jitter (v1.18.14). +- **Surface area**: `agents/main_agent/core/model_config.py` (cache-point placement, `CacheConfig(strategy="auto")` at line 389), `session/turn_based_session_manager.py` (truncation anchor, repeated-compaction path), `session/compaction_models.py` (`cache_ttl_seconds`), plus a read of the system-prompt assembly for any non-deterministic value. +- **Change**: work the cookbook's four applicable techniques as a checklist against code we already have, and **fix what it finds** rather than adding a framework: + 1. **Byte-stable prefix** โ€” grep the system prompt assembly for anything time-, random-, or environment-derived. The cookbook measured a **44% swing from a single `datetime.now()`**. Our `systemPromptHash` should already catch this in prod โ€” so this is as much a test of whether we're *reading* our own instrumentation as of the prompt. + 2. **Layered breakpoints with mixed TTLs** โ€” the cookbook measures **54% cheaper** than a flat point. **Blocked by Strands #3758**: per-section TTLs emit a checkpoint order Bedrock rejects with `ValidationException` on every request. We are not exposed today (we set no per-section TTL), so this is "evaluate and document why not yet", not "adopt". + 3. **Repeated-compaction audit** (from opencode) โ€” does a second compaction pass preserve tool pairing, and does it re-write the prefix each pass? Our own death-spiral incident says this is where the money went. + 4. **Retry cap with jitter** (from opencode) โ€” an uncapped retry on a 424 or throttle re-writes the whole cacheable prefix per attempt. Check whether our retry paths are bounded. +- **Subtracts**: yes โ€” every item is "find waste and delete it". No new abstraction, no new dependency. +- **Effort ร— Impact**: **Low ร— Mediumโ€“High.** The measurement apparatus (`cacheStatus`, fingerprints, `partial_miss`, `wastedUsd`, the cost-anatomy endpoint) already exists โ€” this is applying a known-good checklist to instrumentation we built and then partly stopped reading (three widgets pointed at an empty log group for weeks). +- **Verdict**: **Worth trying** โ€” highest confidence-per-hour item in the list. + +### 5. Resolve the stale review queue before `kaizen-review-prep` consumes it + +- **Source**: internal โ€” `docs/kaizen/review-queue.md` vs. `backend/pyproject.toml` and `gh run list`. Four queue entries assert premises that this week's facts contradict. +- **Surface area**: `docs/kaizen/review-queue.md` only. No code. +- **Change**: resolve four entries as **superseded by events**, with evidence: + - [2026-07-24] / [2026-07-17] / [2026-07-10] `bedrock-agentcore` bump โ†’ **shipped** in #857 (1.9.1 โ†’ **1.21.0**, now zero lag; #482 and #571 closed upstream). The queue's *"we're on 1.9.1, exposed today"* framing is false. Carry forward only the genuinely unfinished residue: **#564 is still open** and wants a local guard. + - [2026-07-24] nightly `DELETE_FAILED` โ†’ **resolved**; Nightly has been green 12 consecutive days (Aug 3โ€“14). + - [2026-07-10] Strands bump โ†’ its own status line already says the bump shipped; split out the still-unadopted capabilities (`continue_on_error`, `Limits` on the headless lane) as their own entries and close the stub. + - Down-rank (don't close) the [2026-07-17] curl-pin item โ€” the `deb13u*` wildcard is holding, so it's correctness debt, not active breakage. +- **Subtracts**: yes โ€” four superseded entries, three of which chain through the same subject and would otherwise be re-ranked as live work. +- **Effort ร— Impact**: **Low ร— Mediumโ€“High.** The impact is entirely leverage: `kaizen-review-prep` fires in ~2 hours against this file. A ranking run that spends its top slot re-recommending a dependency bump we already shipped is worse than no ranking run. +- **Note for review-prep**: this skill does not normally edit `## Resolved` โ€” that move is review-prep's job. +- **Verdict**: โœ… **DONE โ€” executed 2026-08-14 at Phil's request, in this same PR.** + +> **Update (2026-08-14, same day):** Phil asked for this cleanup immediately rather than deferring it to review-prep, so it was executed under explicit instruction. The final count was **nine** entries resolved, not the four first estimated โ€” the audit surfaced a fourth `bedrock-agentcore` bump entry ([2026-07-03]), a second Strands bump entry ([2026-07-03]), a second stale nightly entry ([2026-06-19] `exit 127`), and the [2026-05-29] MCP Apps capability item. +> +> That last one was the most valuable catch and was **not** in the original plan: it asserted `io.modelcontextprotocol/ui` is "spec-canonical", which **this scan could not confirm** โ€” leaving it open risked propagating an unverified identifier straight into an implementation. Residue was carried forward rather than dropped: **#564** (the one failure class the 1.21.0 bump did *not* close) and the **un-adopted Strands capabilities** are now their own `## Open` entries, and the [2026-07-17] curl-pin item was **down-ranked, not closed**. A dangling pointer in an older `## Resolved` entry (which referenced the [2026-07-03] bump as still open) was annotated so the trail stays readable. + +## Take + +Three weeks of accumulation produced an unusually **convergent** window: the entire ecosystem spent it attacking the prompt cache as a design constraint, and we are further along in *measuring* the problem than most of them โ€” `cacheStatus`, fingerprint hashes, `partial_miss`, `wastedUsd` โ€” while being further behind in *escaping* it. Everyone else bought the ability to change the tool set cheaply; we've been paying full price to never change it. That's the gap worth closing, and Idea #1 is a one-day probe that tells us whether it's closeable on Bedrock at all. + +The system is trending **toward** the ecosystem, not away: our `ui_tool_input_partial` is ahead of the reference repo's equivalent, our cost instrumentation independently converged with opencode's, and the seven-week `bedrock-agentcore` bump finally landing puts us at zero lag on the dependency that was our largest standing risk. The counterweight is that we keep building instruments and then not reading them โ€” three dashboard widgets sat pointed at an empty log group for weeks, and AWS quietly moved the span destination again on July 20. The one change Phil would notice first if shipped isn't in the Top 5: **composer draft recovery**, from assistant-ui. We correctly reject a duplicate turn now and still eat the user's typed message. It's an afternoon, and it's the half of 1.14.1 that's missing. + +--- + +## Sources Scanned + +| # | Source | URL | Accessed | Items | +|---|---|---|---|---| +| 1 | AWS What's New (RSS) | https://aws.amazon.com/about-aws/whats-new/recent/feed/ | 2026-08-14 | 5 | +| 2 | AWS ML Blog | https://aws.amazon.com/blogs/machine-learning/ | 2026-08-14 | 1 | +| 3 | AgentCore release notes | https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/release-notes.html | 2026-08-14 | 3 | +| 4 | AgentCore observability config | https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html | 2026-08-14 | 1 | +| 5 | AgentCore Gateway rate limits | https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-rate-limits.html | 2026-08-14 | 1 | +| 6 | Strands SDK releases | https://github.com/strands-agents/sdk-python/releases | 2026-08-14 | 3 | +| 7 | Strands issues (#3348, #3758) | https://github.com/strands-agents/sdk-python/issues | 2026-08-14 | 2 | +| 8 | Strands CHANGELOG.md | https://raw.githubusercontent.com/strands-agents/sdk-python/main/CHANGELOG.md | 2026-08-14 | **404 โ€” file removed** | +| 9 | PyPI strands-agents / -tools | https://pypi.org/project/strands-agents/ | 2026-08-14 | version facts | +| 10 | Reference repo commits | https://github.com/aws-samples/sample-strands-agent-with-agentcore/commits/main | 2026-08-14 | 5 | +| 11 | Reference repo PR #247, #249 | https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/247 | 2026-08-14 | 2 | +| 12 | MCP blog | https://blog.modelcontextprotocol.io | 2026-08-14 | 4 | +| 13 | MCP versioning (spec status) | https://modelcontextprotocol.io/specification/versioning | 2026-08-14 | 1 | +| 14 | MCP Apps extension overview | https://modelcontextprotocol.io/extensions/apps/overview | 2026-08-14 | 1 | +| 15 | MCP Apps API overview | https://apps.extensions.modelcontextprotocol.io/api/documents/Overview.html | 2026-08-14 | 1 | +| 16 | FastMCP releases | https://github.com/jlowin/fastmcp/releases | 2026-08-14 | 4 | +| 17 | PyPI fastmcp | https://pypi.org/project/fastmcp/ | 2026-08-14 | version facts | +| 18 | Vercel AI SDK โ€” MCP Apps | https://ai-sdk.dev/docs/ai-sdk-core/mcp-apps | 2026-08-14 | 1 | +| 19 | Vercel AI SDK โ€” tool approvals | https://ai-sdk.dev/docs/agents/tool-approvals | 2026-08-14 | 1 | +| 20 | assistant-ui releases | https://github.com/Yonom/assistant-ui/releases | 2026-08-14 | 2 | +| 21 | NN/g โ€” PROVE framework | https://www.nngroup.com/articles/prove-framework/ | 2026-08-14 | 1 | +| 22 | Anthropic news / Opus 5 | https://www.anthropic.com/news/claude-opus-5 | 2026-08-14 | 2 | +| 23 | OpenAI API changelog | https://developers.openai.com/api/docs/changelog | 2026-08-14 | 3 | +| 24 | Google DeepMind blog | https://blog.google/technology/google-deepmind/ | 2026-08-14 | 0 (thin) | +| 25 | Anthropic engineering blog | https://www.anthropic.com/engineering | 2026-08-14 | **0 โ€” no posts in window** | +| 26 | Claude Code CHANGELOG | https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md | 2026-08-14 | 3 | +| 27 | pydantic-ai releases | https://github.com/pydantic/pydantic-ai/releases | 2026-08-14 | 2 | +| 28 | opencode releases | https://github.com/anomalyco/opencode/releases | 2026-08-14 | 3 | +| 29 | LibreChat releases | https://github.com/danny-avila/LibreChat/releases | 2026-08-14 | 3 | +| 30 | Bedrock pricing | https://aws.amazon.com/bedrock/pricing/ | 2026-08-14 | **could not extract Claude rows** | +| 31 | AgentCore pricing | https://aws.amazon.com/bedrock/agentcore/pricing/ | 2026-08-14 | 2 | +| 32 | bedrock-agentcore SDK issues + releases | https://github.com/aws/bedrock-agentcore-sdk-python/issues | 2026-08-14 | 4 | +| 33 | starter-toolkit issues | https://github.com/aws/bedrock-agentcore-starter-toolkit/issues | 2026-08-14 | 1 | +| 34 | HN (Algolia API) | https://news.ycombinator.com/ | 2026-08-14 | 4 | +| 35 | Reddit (search snippets only) | โ€” | 2026-08-14 | **0 โ€” domain blocked for fetch** | +| 36 | Anthropic cookbook | https://github.com/anthropics/claude-cookbooks/commits/main | 2026-08-14 | 3 | +| 37 | PyPI / npm registries (version pins) | https://pypi.org ยท https://registry.npmjs.org | 2026-08-14 | 10 rows | + +## Web Budget + +**Used: ~71 / 50 requests** (subagent-reported tool calls; ~80 at the HTTP level, since the version-pin agent issued 18 registry fetches across 9 calls). + +**Overage justification**: the scan window was **21 days, not 7** โ€” two Fridays were missed, so three weeks of AWS What's New, three weeks of Strands/FastMCP/reference-repo releases, and a *finalized MCP spec revision* all landed in one pass. The categories that ran over were the ones carrying real signal: MCP spec status (6 โ€” four specific SEP outcomes needed confirming), Strands (7 โ€” two named issues each needed their own page), agentic UI/UX (7 โ€” two source URLs 404'd and were recovered by search), and version pins (9 calls / 18 fetches โ€” every date re-verified against raw registry JSON after a summarizer fabricated one). + +**Skipped (unreachable / blocked)**: +- `reddit.com` โ€” domain-blocked for fetch (consistent with the [2026-05-18] decision); search snippets only, and nothing dated in-window surfaced. +- `openai.com/news` โ€” **HTTP 403**; substituted with https://developers.openai.com/api/docs/changelog, which is the better source anyway (dated entries). +- `strands-agents/sdk-python/CHANGELOG.md` โ€” **404, the file no longer exists** on `main`. +- `nngroup.com/topic/artificial-intelligence/` โ€” 404; correct path is `/topic/ai/`. +- `ai-sdk.dev/docs/ai-sdk-ui/mcp-apps` โ€” 404; recovered at `/docs/ai-sdk-core/mcp-apps`. + +**Skipped (budget)**: +- `modelcontextprotocol/servers` (new/retired servers) โ€” MCP budget exhausted on spec-status confirmation, which was the higher-value question. +- LangChain and LlamaIndex release notes โ€” deprioritized behind Claude Code and pydantic-ai. +- Linear and Cursor product blogs โ€” not reached within the UI/UX budget; **the obvious gap to cover first next week.** +- FastMCP issues tracker โ€” the releases page was sufficient. +- `ext-apps` releases โ€” fetched but returned stale 2025-dated content conflicting with the Apps API docs; **not treated as reliable and not cited.** + +**Fabrication caught**: a summarizer returned an invented PyPI upload date for `fastapi` (`2024-12-19T17:28:47.123456Z`); the true date is 2026-07-29. Every version date in this doc was subsequently re-verified against raw registry JSON. Worth remembering as a standing hazard for this skill. diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md index 019da669c..1cf3c4189 100644 --- a/docs/kaizen/review-queue.md +++ b/docs/kaizen/review-queue.md @@ -5,12 +5,45 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Open -### [2026-07-24] Bump `bedrock-agentcore` 1.9.1 โ†’ 1.18.1 (closes #482 SSE deadlock + #571 Memory reorder; + security patches) -- **Source**: research/2026-07-24.md โ–ธ Top 5 #1 โ€” bedrock-agentcore 1.18.1 (Jul 17, security patch over 1.18.0's #571 fix; #482 fix in 1.17.0) โ€” https://github.com/aws/bedrock-agentcore-sdk-python/releases -- **Surface**: backend (`backend/pyproject.toml` + coupled `boto3`; inference-api chat router + `TurnBasedSessionManager` flush ordering; full local pytest suite) +> โœ… **Queue hygiene completed 2026-08-14** (at Phil's request, ahead of `kaizen-review-prep`). **Nine** stale entries were resolved: four `bedrock-agentcore` bump entries and two Strands bump entries (all **shipped** in #857 โ€” `bedrock-agentcore` 1.9.1 โ†’ **1.21.0** at zero lag, `strands-agents` โ†’ **1.51.0**; #482 and #571 closed upstream), two nightly-CI entries (**green 12 consecutive days**), and two MCP Apps spec-prep entries (superseded now the 2026-07-28 spec is final). Genuine residue was carried forward, not dropped: **#564** (still open upstream) and the **un-adopted Strands capabilities** are now their own entries below. See `## Resolved` for the evidence trail. + +### [2026-08-14] Probe Anthropic's mid-conversation tool-mutation beta on Bedrock +- **Source**: research/2026-08-14.md โ–ธ Top 5 #1 โ€” https://www.anthropic.com/news/claude-opus-5 (2026-07-24): "change which tools Claude can use mid-conversation without invalidating the prompt cache" (beta). Convergent: pydantic-ai `ToolReturn.tools`/`load_capability` (v2.26.0), MCP SEP-2549 cacheable list results, Claude Code 2.1.232 fork-inherits-cache. +- **Surface**: backend (`agents/main_agent/core/model_config.py` Bedrock request construction; `_build_filtered_tools`; the `toolConfigHash` fingerprint on `C#` rows; `GET /admin/costs/sessions/{id}/calls`) +- **Effort ร— Impact**: L (probe) ร— H +- **Subtracts**: conditionally, and enormously โ€” could narrow the prompt-cache contract's most expensive clause ("any list reaching `toolConfig` must be deterministically ordered at its source") and unblock three separately-queued items that share the same wall +- **Unlocks**: per-turn tool filtering (the real fix for MCP tool-list bloat); `@`-mention switches that don't structurally re-write a 30kโ€“150k-token prefix; the Vercel "app-visible-only tools" and pydantic-ai tool-search patterns +- **Status**: open โ€” **recommended #1.** A probe, not a build: (a) does Bedrock Converse expose the beta at all, or is it Claude-API-only? (b) what's the header/param, and does `strands` pass it through? (c) measured on a real session, does adding a tool mid-conversation leave `cacheReadInputTokens` intact? **A negative result is a valid outcome** โ€” same shape as G1/G3, and it closes an assumption currently sitting under several queued items. + +### [2026-08-14] Migrate the MCP Apps host off `initialize`/`serverInfo` to `server/discover` +- **Source**: research/2026-08-14.md โ–ธ Top 5 #2 โ€” MCP **2026-07-28 is now the Current protocol version**; SEP-2575 removed the `initialize`/`initialized` handshake + `Mcp-Session-Id` โ€” https://modelcontextprotocol.io/specification/versioning ยท https://blog.modelcontextprotocol.io/posts/2026-07-28/ +- **Surface**: backend (`agents/main_agent/integrations/mcp_apps.py:673` โ€” the `getattr(result, "serverInfo", None)` capture; `_mcp_apps_server_info` consumers ~L454โ€“461 / L628โ€“635; `streaming/stream_coordinator.py:1680` `ui_resource` header emission; the `ClientCapabilities(experimental=...)` subclassing at `mcp_apps.py:26`) +- **Effort ร— Impact**: M ร— Mโ€“H +- **Subtracts**: yes โ€” retires the `initialize`-response dependency, and collapses the "fresh MCP session per call" concern behind the MCP Apps proxy-call 504 work (there is no protocol-level session left to preserve) +- **Unlocks**: conformance with a published host matrix (Claude, VS Code Copilot, M365 Copilot, Goose, Postman); readiness for **MRTR (SEP-2322)** โ€” the sanctioned interrupt/resume shape, which would let OAuth consent and tool approvals resume *without* holding an SSE stream open against the 600s timeout; readiness for SEP-2243 header-based Gateway routing +- **Status**: open โ€” **SUPERSEDES the [2026-07-24] "prep the MCP Apps host for the 2026-07-28 spec" item** (written when the spec was an RC; it is now final). **Two cheap verifications required before any code**: (a) confirm the UI capability identifier against the apps **spec source** โ€” this scan could NOT confirm `io.modelcontextprotocol/ui` vs `experimental.ui`; (b) confirm `ui/notifications/tool-input-partial` still exists in the spec (absent from the SDK overview page โ€” not evidence of removal, but our `ui_tool_input_partial` relay depends on it). Keep the handshake path as a compatibility branch: `server/discover` is **mandatory for servers, optional for clients**. + +### [2026-08-14] Attack the W5 memory bill โ€” self-managed LTM strategies + model Runtime Instances +- **Source**: research/2026-08-14.md โ–ธ Top 5 #3 โ€” https://aws.amazon.com/bedrock/agentcore/pricing/ (built-in long-term strategies **$0.75/1,000 records/month** vs override/self-managed **$0.25/1,000**) + Runtime **Instances** GA (EC2 + 12% fee, **Savings Plans / ODCR eligible**, 14-day sessions) โ€” https://aws.amazon.com/about-aws/whats-new/2026/08/aws-bedrock-agentcore-runtime-instances-generally-available/. **Verified internally**: `infrastructure/lib/constructs/agentcore/memory-construct.ts:77` configures all three built-in strategies. +- **Surface**: infrastructure / backend (`memory-construct.ts` `memoryStrategies` array + `eventExpiryDuration: 90`; `inference-agentcore-construct.ts` compute type + memory allocation; `apis/app_api/memory/routes.py` `/facts/` `/preferences/` `/summaries/` readers) - **Effort ร— Impact**: M ร— H -- **Subtracts**: yes โ€” retires the queued [2026-05-22] hand-written #482 guard (library-native); the #571 ordering fix complements (not replaces) `_repair_tool_pairing` -- **Status**: open โ€” **highest priority; now SEVEN weeks queued while the release keeps advancing (1.18.1).** SUPERSEDES the [2026-07-17] "โ†’ 1.18.0" item (retarget 1.18.1). We're on 1.9.1, exposed to #482 + #571 today. Validate ms event-flooring vs our flush ordering; #564 (eventual-consistency read gap) stays unfixed โ€” keep agent-cache continuity primary. **Sequence AFTER Nightly is green (the [2026-07-24] nightly-stack item below) so the dep-bump gate can vouch for it.** +- **Subtracts**: yes (track 1) โ€” a **3ร— per-record price cut** on a line item we're paying the premium tier for by default, with no deliberate decision behind it +- **Unlocks**: Savings-Plans/ODCR-covered agent compute โ€” the first commitment-discount lever AgentCore has ever offered โ€” and 14-day persistent sessions, which changes the idle-reaper calculus (#827) +- **Status**: open โ€” **the named W5 gap, and the first week the ecosystem handed us a real lever on it.** Split into two tracks and ship track 1's *measurement* first: (1) determine per-strategy record volume and whether each is *read* often enough to justify 3ร— the self-managed rate; (2) model Instances against real dev/prod session-concurrency โ€” Instances favour many concurrent short sessions sharing an agent, penalize spiky low-utilization. โš ๏ธ **AWS's own Instances HN post drew 1 point / 0 comments โ€” zero independent validation.** Model it against our numbers; do not adopt on the pitch. + +### [2026-08-14] Run the Anthropic `cost_optimization` cookbook as an audit against our own contract +- **Source**: research/2026-08-14.md โ–ธ Top 5 #4 โ€” https://github.com/anthropics/claude-cookbooks/blob/main/cost_optimization/cost_optimization.ipynb (2026-08-12), seven measured strategies + a `usage_cost()` helper. Reinforced by opencode v1.18.17 (turn-aligned compaction) and v1.18.14 (retry cap with jitter). +- **Surface**: backend (`core/model_config.py` cache-point placement + `CacheConfig(strategy="auto")` at L389; `session/turn_based_session_manager.py` truncation anchor + repeated-compaction path; `session/compaction_models.py` `cache_ttl_seconds`; system-prompt assembly) +- **Effort ร— Impact**: L ร— Mโ€“H +- **Subtracts**: yes โ€” every item is "find waste and delete it"; no new abstraction, no new dependency +- **Status**: open โ€” **highest confidence-per-hour item in the scan.** Four checks: (1) **byte-stable prefix** โ€” grep system-prompt assembly for time/random/env-derived values (the cookbook measured a **44% swing from one `datetime.now()`**; this also tests whether we're *reading* `systemPromptHash`); (2) **layered mixed-TTL breakpoints** โ€” 54% cheaper upstream, but โš ๏ธ **blocked by Strands #3758** (per-section TTLs emit a checkpoint order Bedrock rejects with `ValidationException` on every request โ€” we set no per-section TTL today, so evaluate + document, don't adopt); (3) **repeated-compaction audit** โ€” does pass 2 preserve tool pairing and avoid re-writing the prefix? (our death-spiral incident says this is where the money went); (4) **retry cap with jitter** โ€” an uncapped retry on a 424/throttle re-writes the whole prefix per attempt. + +### [2026-08-14] Guard against `bedrock-agentcore` #564 โ€” the one failure class the 1.21.0 bump did NOT close +- **Source**: research/2026-08-14.md โ–ธ Community + GitHub issues; the carried-forward residue of the four now-resolved `bedrock-agentcore` bump entries โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/564 (**still open**; #482 and #571 closed, #564 did not) +- **Surface**: backend (`agents/main_agent/session/turn_based_session_manager.py` restore path; `AgentCoreMemorySessionManager` `read_agent`/`read_session` marker-event lookup) +- **Effort ร— Impact**: M ร— Mโ€“H +- **Subtracts**: no โ€” a local guard against an upstream gap; delete it if/when #564 is fixed upstream +- **Status**: open โ€” **the bump's value was real but incomplete; this is what's left.** Metadata-filtered `ListEvents` transiently misses marker events, so the manager treats the turn as a **new session** and skips history restoration *even though the data exists in the unfiltered view*. Cost impact precedes correctness impact: a false "new session" both loses context **and** re-writes the entire cacheable prefix at the cache-write premium. Related upstream risks worth checking in the same pass: **#621** (`filter_restored_tool_context` incompatible with extended thinking โ€” same restore path) and **#629** (TracerProvider never flushed before microVM freeze, so end-of-invocation spans are dropped โ€” which means our own cost telemetry under-reports the final turn of every session). ### [2026-07-24] Add GPT-5.6 Terra + Luna to the model catalog via the existing Mantle Responses path - **Source**: research/2026-07-24.md โ–ธ Top 5 #2 โ€” GPT-5.6 Sol/Terra/Luna GA on Bedrock via Mantle (confirms last week's flagged-for-verification item) โ€” https://aws.amazon.com/about-aws/whats-new/2026/07/openai-gpt-sol-terra/ @@ -28,21 +61,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: captures GPT-5.6's 90% cached-input discount instead of leaving it on the table - **Status**: open โ€” SUPERSEDES the [2026-07-17] caching-audit item (adds the GPT-5.6 explicit-breakpoint motivation). Instrument `cache_read`/`cache_write` per provider (1.9.0 observability surfaces them); wire explicit breakpoints on the Mantle leg; confirm the Bedrock manual cache-point still engages post-1.48; do NOT switch to `strategy="auto"`. Coupled to the GPT-5.6 item above. -### [2026-07-24] Prep the MCP Apps host for the 2026-07-28 spec (`serverInfo` โ†’ `server/discover`; SEP-2575 handshake removal) -- **Source**: research/2026-07-24.md โ–ธ Top 5 #4 โ€” MCP 07-28 RC final in 4 days; SEP-2575 removes the `initialize`/`initialized` handshake + `Mcp-Session-Id`, adds `server/discover`; MCP Apps (SEP-1865) graduates to a first-class official extension โ€” https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/ -- **Surface**: backend (inference-api MCP Apps host โ€” the `initialize` `serverInfo` read that resolves App-frame `serverName`/`icon`; capability advertisement `experimental.ui` โ†’ `io.modelcontextprotocol/ui`; `ui_resource` SSE header path; existing `manifest.json`-fallback icon resolution) -- **Effort ร— Impact**: M ร— Mโ€“H -- **Subtracts**: yes โ€” retires the pre-standard `initialize`-serverInfo dependency + the `experimental.ui` id -- **Unlocks**: RC-conformant negotiation with spec-final MCP hosts/servers; readiness for SEP-2567 stateless transport + SEP-2322 multi-round-trip elicitation (the App user-input flow) -- **Status**: open โ€” **near-term (spec-final in 4 days); ABSORBS the [2026-05-29] "align MCP Apps capability advertisement" item and sharpens it with the SEP-2575 handshake-removal detail.** Migrate `serverName`/`icon` off `serverInfo` to `server/discover` (manifest.json fallback stays); confirm final-spec details next Friday before building (don't build against a moving RC). - -### [2026-07-24] Fix the nightly `DELETE_FAILED` stuck ephemeral stack + make teardown resilient -- **Source**: research/2026-07-24.md โ–ธ Top 5 #5 โ€” internal friction: Nightly red Jul 23โ€“24 (run 30083857845), `nightly-develop-PlatformStack is in DELETE_FAILED state and can not be updated`; green Jul 14โ€“17 before (a new teardown-idempotency failure class, distinct from the June exit-127 + July curl-pin clusters) -- **Surface**: infra/CI (`.github/workflows/nightly-deploy-pipeline.yml` ephemeral deploy/teardown lane + PlatformStack `RemovalPolicy`/`autoDeleteObjects` in the nightly context โ€” retained buckets/log-groups/custom resources are the usual `DELETE_FAILED` culprits) -- **Effort ร— Impact**: L ร— Mโ€“H -- **Subtracts**: yes โ€” removes a recurring nightly-wedge class; restores the dep-bump safety gate -- **Status**: open โ€” **cheapest win with outsized leverage: unblocks the #1 keystone bump's safety net.** (1) immediate โ€” manually force-delete the wedged stack (skip the un-deletable resource) so Nightly goes green; (2) durable โ€” pre-deploy step that force-deletes a leftover `DELETE_FAILED`/`ROLLBACK_COMPLETE` stack of that name before re-creating + nightly-context removal policies. - ### [2026-07-19] Track harness-sdk#3348 (rolling pair of message cachePoints) โ€” local workaround gated on dashboard evidence - **Source**: Phil-initiated (PR #697 follow-up) โ€” https://github.com/strands-agents/harness-sdk/issues/3348 (filed by philmerrell, open, no maintainer response yet); prod session aecd387d (18-way parallel tool fan-out โ†’ cacheRead=0 / cacheWrite=134k mid-turn, the ~20-block Anthropic lookback miss mode documented in `model_config.py:366`) - **Surface**: backend (`agents/main_agent/core/model_config.py` 3-cachePoint budget). If built locally: strands 1.48's `_inject_cache_point` **strips any pre-existing message-level cachePoints**, so a rolling pair requires dropping `CacheConfig(strategy="auto")` and hand-placing both points via a hook โ€” the 4th Bedrock cachePoint slot is free. Position tests in `tests/agents/main_agent/core/test_bedrock_cache_points.py` are the safety net. @@ -57,20 +75,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Subtracts**: partial โ€” bounds MCP/tool payload growth at the source instead of relying solely on reactive below-anchor truncation in `TurnBasedSessionManager` (which stays for legacy history) - **Status**: open โ€” spike before commitment; four known gotchas: (1) `evict_after_cycles=20` runs on `BeforeModelCallEvent` and touches *prior* messages โ€” potential byte-stability cache-buster, verify semantics or set `None` and expire via S3 lifecycle; (2) `model.count_tokens` per tool result adds latency, and Bedrock CountTokens rejects `us.*` inference-profile ids (de-prefix precedent in context attribution); (3) adoption flips `toolConfigHash` once (expected; the new tool must land in the deterministically-ordered tool list); (4) check SPA tool-result rendering against placeholder content. -### [2026-07-17] Bump `bedrock-agentcore` 1.9.1 โ†’ 1.18.0 (closes #571 cross-process Memory reorder + #482 SSE deadlock) -- **Source**: research/2026-07-17.md โ–ธ Top 5 #1 โ€” bedrock-agentcore 1.18.0 (Jul 10, PR #572/#573 close #571; #482 fix in 1.17.0/PR #563) โ€” https://github.com/aws/bedrock-agentcore-sdk-python/releases -- **Surface**: backend (`backend/pyproject.toml` + coupled `boto3`; inference-api chat router + `TurnBasedSessionManager` flush ordering; full local pytest suite) -- **Effort ร— Impact**: M ร— H -- **Subtracts**: yes โ€” retires the queued [2026-05-22] hand-written #482 guard (library-native); the #571 ordering fix complements (not replaces) the just-shipped `_repair_tool_pairing` -- **Status**: open โ€” **highest priority; the release we've queued for six weeks now exists and landed in-window.** SUPERSEDES the [2026-07-03] "โ†’ 1.17.0" item AND the [2026-07-10] "double-forced" item below (which said "bump to 1.17.0, track 1.18 for #571" โ€” 1.18.0 now exists with the #571 fix, so retarget 1.18.0 and consolidate the two into this one). We're on 1.9.1, ~9 minors behind, exposed to both #571 and #482 while Scheduled Runs + Memory Spaces + conversation-sharing lean hard on Memory. Validate ms event-flooring vs our flush ordering; #564 (eventual-consistency read gap) stays unfixed โ€” keep agent-cache continuity primary. - -### [2026-07-17] Audit multi-provider prompt caching (issue #642 + Strands #3144) -- **Source**: research/2026-07-17.md โ–ธ Top 5 #2 โ€” internal issue #642 (Jul 11); Strands #3144 (`CacheConfig(strategy="auto")` never caches system prompt, fix PR #3145 open) โ€” https://github.com/strands-agents/sdk-python/issues/3144 -- **Surface**: backend (`to_bedrock_config` cache-point injection; Mantle Responses builder `build_mantle_model`/`_create_mantle_model`; OpenAI-compatible path; `CountTokensBedrockModel` cache-token read) -- **Effort ร— Impact**: Lโ€“M ร— Mโ€“H -- **Subtracts**: yes โ€” consolidates per-provider cache logic; kills a silent full-input-token cost regression if a Mantle/OpenAI path caches nothing -- **Status**: open โ€” a filed internal tech-debt issue with a library-confirmed failure mode. Instrument `cache_read`/`cache_write` ratios per provider (Strands 1.46 surfaces them); confirm the Bedrock manual cache-point still engages post-1.47 and do NOT switch to `strategy="auto"` expecting system-prompt caching. Strongest internal-signal fit after the bump. - ### [2026-07-17] Adopt Strands `Limits` on the unattended Scheduled Runs / headless lane - **Source**: research/2026-07-17.md โ–ธ Top 5 #3 โ€” convergent harness rail (Claude Code 2.1.212 spawn cap + opencode 1.18.2 `subagent_depth`); Strands `Limits` now available (we're on 1.47). - **Surface**: backend (headless/scheduled path `apis/shared/harness/run_agent_headless` per [2026-07-06] managed-Harness spike + agent-loop per-invocation config) @@ -91,30 +95,15 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Surface**: infra/CI (`backend/Dockerfile.app-api` + `Dockerfile.inference-api` line 42 `apt-get install curl=...`; check `scheduled-runs`/`kb-sync`) - **Effort ร— Impact**: L ร— Mโ€“H - **Subtracts**: yes โ€” removes a recurring deploy-breaker class; the `deb13u*` wildcard only defers the next break to a Debian series/base-image bump -- **Status**: open โ€” replace the version-pinned curl with unpinned (latest security patch) or the base image's `curl-minimal` (as the Lambda images already do). The pin was never a supply-chain control โ€” it's a HEALTHCHECK runtime probe. Cheapest durable win; the band-aid will bite again. +- **Status**: open โ€” **DOWN-RANKED 2026-08-14: correctness debt, not active breakage.** The `deb13u*` wildcard has held all window (Backend Deploy + Nightly green Aug 3โ€“14), so the urgency claim in the original entry no longer applies. The fix is unchanged and still right: replace the version-pinned curl with unpinned (latest security patch) or the base image's `curl-minimal` (as the Lambda images already do). The pin was never a supply-chain control โ€” it's a HEALTHCHECK runtime probe. The wildcard only defers the next break to a Debian series / base-image bump. -### [2026-07-10] Bump `bedrock-agentcore` off 1.9.1 โ€” now double-forced (#482 SSE deadlock + NEW #571 Memory-reorder) -- **Source**: research/2026-07-10.md โ–ธ Top 5 #1 โ€” https://github.com/aws/bedrock-agentcore-sdk-python/pull/563 (#482, in 1.17.0) + **NEW** https://github.com/aws/bedrock-agentcore-sdk-python/issues/571 (cross-process Memory event-reorder corruption, fix pending a post-1.17.0 release). -- **Surface**: backend (`backend/pyproject.toml`, inference-api chat router, `AgentCoreMemorySessionManager` usage; full local pytest suite) -- **Effort ร— Impact**: M ร— H -- **Subtracts**: yes โ€” retires the queued [2026-05-22] #482 hand-written guard (library-native subtraction) -- **Status**: open โ€” **CONSOLIDATED into the [2026-07-17] "โ†’ 1.18.0" item above** (the #571 fix this entry was tracking landed in 1.18.0 on Jul 10 โ€” the post-1.17.0 release it awaited). Treat as one item; review-prep should resolve this stub. 1.1.0/1.2.0 shipped Scheduled Runs (multi-replica Runtime) + Memory Spaces (heavier Memory use), amplifying exactly the failure classes #482/#571 corrupt. - -### [2026-07-10] Bump Strands 1.40 โ†’ 1.47; adopt `continue_on_error` MCP resilience (#3101) + hook ordering -- **Source**: research/2026-07-10.md โ–ธ Top 5 #2 โ€” Strands 1.46โ€“1.47 (https://github.com/strands-agents/sdk-python/releases). **Supersedes the [2026-07-03] 1.45 item** (now 7 minors behind). -- **Surface**: backend (`agents/main_agent/` hooks + `to_bedrock_config` + compaction, `FilteredMCPClient`/gateway targets, `CountTokensBedrockModel`) -- **Effort ร— Impact**: Mโ€“H ร— H -- **Subtracts**: candidate โ€” hand-rolled MCP-abort handling (`continue_on_error`), custom cache-point plumbing (`cache_tools_ttl`), runaway guard (`Limits`) -- **Unlocks**: a flaky external/Gateway MCP server no longer aborts the turn (newly relevant โ€” Scheduled Runs use external tools unattended); `Limits` per-invocation cost caps; deterministic hook ordering (the enabler for the tool-approval fix) -- **Status**: open โ€” **the bump itself SHIPPED** (commit `42e69bc7` "upgrade Strands to 1.47.0"; the pin is now 1.47.0). Remaining follow-on work is separately queued: `Limits` adoption on the headless lane is the [2026-07-17] item above; `continue_on_error` on the MCP client + optional hook ordering (#2559) + the `cache_tools_ttl`/`context_manager="auto"` audits are still open here (decisions.md 2026-05-18 bars a bare compaction swap). Review-prep should split "bump = done" from the un-adopted capabilities. - -### [2026-07-10] Audit whether prompt caching actually engages in `to_bedrock_config` (Strands #3144) -- **Source**: research/2026-07-10.md โ–ธ Top 5 #3 โ€” **NEW** Strands open issue #3144 (`CacheConfig(strategy="auto")` never caches the system prompt); Strands 1.46 now surfaces `cache_read`/`cache_write` tokens in the metadata chunk (#2302). https://github.com/strands-agents/sdk-python/issues -- **Surface**: backend (`agents/main_agent/` cache-point config in `to_bedrock_config`; the metadata/usage path feeding `CountTokensBedrockModel` + the context-attribution badge) -- **Effort ร— Impact**: Lโ€“M ร— Mโ€“H -- **Subtracts**: no โ€” a cost-correctness audit (may confirm we're fine, or expose a silent full-input-token regression) -- **Unlocks**: potentially large per-turn cost cut if caching is silently off; a verifiable caching invariant -- **Status**: open โ€” **subsumed by / merge with the [2026-07-17] "multi-provider prompt caching" item above** (same Bedrock cache-point surface + Strands #3144, extended to the Mantle/OpenAI provider shapes). Assert cache points are written *and* read via `cache_read`/`cache_write` counts; measurement is nearly free now that 1.46 surfaces the tokens. +### [2026-08-14] Adopt the Strands capabilities the 1.51 bump made available but did not wire +- **Source**: research/2026-08-14.md โ–ธ queue-hygiene cleanup โ€” the split-out residue of the [2026-07-10] "Strands 1.40 โ†’ 1.47" entry, whose *bump* half shipped (now pinned **1.51.0** via #857). Capability refs: `continue_on_error` MCP resilience (#3101), optional hook ordering (#2559), `cache_tools_ttl`, `context_manager="auto"` โ€” https://github.com/strands-agents/sdk-python/releases +- **Surface**: backend (`agents/main_agent/` hooks + `to_bedrock_config` + compaction; `FilteredMCPClient`/gateway targets; `CountTokensBedrockModel`) +- **Effort ร— Impact**: M ร— Mโ€“H +- **Subtracts**: candidate โ€” hand-rolled MCP-abort handling (`continue_on_error`) and custom cache-point plumbing (`cache_tools_ttl`) are both library-native replacements +- **Unlocks**: a flaky external/Gateway MCP server no longer aborts the turn (load-bearing โ€” Scheduled Runs use external tools unattended); deterministic hook ordering, the enabler for the tool-approval fix +- **Status**: open โ€” **the bump is DONE; this is only the un-adopted capability list.** `Limits` on the headless lane is tracked separately at [2026-07-17]. โš ๏ธ `context_manager="auto"` is **barred as a bare swap** by decisions.md 2026-05-18 โ€” our compaction additionally does tool-content truncation, LTM summary retrieval, and DynamoDB checkpoint persistence, and drives the `compaction` SSE event; only a migration design covering all four is in scope. Also re-check `cache_tools_ttl` against Strands **#3758** (per-section TTLs can emit a checkpoint order Bedrock rejects on every request) before wiring it. ### [2026-07-10] Tool-approval policy layer + signed approvals (Vercel AI SDK) โ€” evolve the queued approval item - **Source**: research/2026-07-10.md โ–ธ Top 5 #4 โ€” Vercel AI SDK tool-approvals (https://ai-sdk.dev/docs/agents/tool-approvals). Builds on the [2026-07-03] tool-approval item. @@ -148,21 +137,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: Mantle-first capability gradient (Responses API, server-side tool use, async/long-running, Projects/Workspaces) once Claude parity lands; a uniform OpenAI-compatible lane for non-Claude models inside Bedrock without a second vendor SDK. - **Status**: open โ€” **strategic/future-proofing, not urgent; recommend Defer (watchlist) + a small non-Claude-lane spike.** Corrected findings: (1) `bedrock-runtime` is "fully supported," no EOL signal โ€” Mantle recommendation is greenfield-onboarding language, but the capability gradient toward Mantle is real. (2) **The "persisted Converse wire shape = multi-model lock-in" concern does NOT hold** โ€” verified `strands/types/content.py:78` `ContentBlock` (`toolUse`/`toolResult`/`reasoningContent`) IS Strands' provider-neutral canonical shape; every Strands provider round-trips it to/from Anthropic Messages / OpenAI Chat Completions. AgentCore Memory abstracts *persistence*; Strands abstracts *multi-model shape*. Switching a provider's endpoint changes Strands `format_request` internals, **not** our persistence schema or `_convert_content_block`. No schema-decoupling PR needed. (3) The real bedrock-runtime ties are narrow: the 3 direct-Converse bypasses, `CountTokens` (no Mantle equal โ€” powers context-attribution + compaction), and cross-region profiles (`us.*`/`global.*`, Mantle-absent). (4) **Do NOT migrate the Claude chat path to Mantle yet**: Opus 4.8 on Mantle is Messages-API-only (Chat Completions/Responses = No), so Mantle's headline built-ins don't apply to our primary model; Mantle also lacks cross-region inference, native `CountTokens`, structured outputs, and **Guardrails** (runtime-only โ€” see [2026-06-19] Guardrails item, which this reinforces). Pricing identical; Mantle default TPM not a win (`20M in/4M out` vs runtime `30M`). **Reopen trigger:** a Strands Anthropic-Messages-on-Mantle provider ships **AND** cross-region + native token counting reach Claude-on-Mantle. Interim, low-risk value = finishing the non-Claude OpenAI-compatible lane already scaffolded in `_create_mantle_model()`. -### [2026-07-03] Bump `bedrock-agentcore` 1.9.1 โ†’ 1.17.0 (closes SSE-deadlock #482) -- **Source**: research/2026-07-03.md โ–ธ Top 5 #1 โ€” https://github.com/aws/bedrock-agentcore-sdk-python/pull/563 (issue #482); internal inference-api SSE-over-Runtime exposure. -- **Surface**: backend (`backend/pyproject.toml`, inference-api chat router; full local pytest suite โ€” the only correctness gate, pytest isn't in CI) -- **Effort ร— Impact**: M ร— H -- **Subtracts**: yes โ€” retires the queued [2026-05-22] "defensive guard against #482" work item; the fix (`put_nowait` + disconnect stop-event + source `aclose()`) is now upstream (library-native subtraction) -- **Status**: open โ€” **highest priority; we're on 1.9.1 and exposed today** to a silent, process-wide container hang that keeps `/ping` green when an SSE consumer stops draining. This supersedes/absorbs the queued [2026-05-22] guard item. - -### [2026-07-03] Bump Strands 1.40 โ†’ 1.45 + adopt hook ordering; audit cache_tools_ttl / context_manager / Limits -- **Source**: research/2026-07-03.md โ–ธ Top 5 #2 โ€” Strands releases 1.41โ€“1.45 (https://github.com/strands-agents/sdk-python/releases, now `harness-sdk` monorepo). -- **Surface**: backend (`backend/src/agents/main_agent/` hooks + BedrockModel config + compaction, `to_bedrock_config`, `CountTokensBedrockModel`) -- **Effort ร— Impact**: Mโ€“H ร— H -- **Subtracts**: candidate โ€” custom cache-point plumbing (`cache_tools_ttl`, 1.41) and possible compaction simplification (`context_manager="auto"`, 1.43 โ€” gated on the SSE-contract check per decisions.md 2026-05-18, not a bare drop-in) -- **Unlocks**: `Limits` per-invocation token/cost caps (first-class budget guard we lack) -- **Status**: open โ€” **supersedes the queued [2026-06-19] "1.40 โ†’ 1.44" item.** Adopt optional hook ordering (#2559) to deterministically sequence the OAuth-consent + tool-approval BeforeToolCall hooks through the tool-fold. Only breaking change 1.41โ†’1.45 is Mistral (N/A). Run the full local pytest suite; watch compaction + count_tokens/context-attribution. - ### [2026-07-03] Model-settings refresh: reinstate Fable 5 + add Sonnet 5 with temperature-suppression guard - **Source**: research/2026-07-03.md โ–ธ Top 5 #3 โ€” Fable 5 reinstated (https://aws.amazon.com/blogs/aws/anthropic-claude-fable-5-on-aws-mythos-class-capabilities-with-built-in-safeguards-now-available/); Sonnet 5 GA + promo pricing (https://aws.amazon.com/bedrock/pricing/); ref-repo `NO_TEMPERATURE_MODELS` (commit 35bc3a9). - **Surface**: cross-cutting (inference-api model config + model-settings admin, `to_bedrock_config`, `CountTokensBedrockModel` de-prefix, frontend model picker) @@ -195,13 +169,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: deployers attach content-safety filtering + staff-alerting monitoring to all model invocations without modifying inference-api source (FERPA duty-of-care for higher-ed: proactive self-harm/crisis-language monitoring Claude's reactive layer doesn't surface) - **Status**: open โ€” strongest fit (filed issue + library-native path). **Decide in-agent vs. gateway-level in one pass** โ€” the [2026-07-03] "gateway-level Guardrails (AgentCore Policy)" item folds into this #480 decision (one gateway policy blankets every MCP target, model-independent). Verify guardrail *resource* region availability + SSE streaming-mode compatibility. Reviewed reviews/2026-07-03.md โ–ธ Proposal #4. -### [2026-06-19] Fix Nightly Build & Test (`exit 127` at install โ€” ~14 consecutive failures) -- **Source**: research/2026-06-19.md โ–ธ Internal Audit + Top 5 #2 โ€” `gh run view 27820449858 --log-failed` shows `exit code 127` on every install/setup step (June 19); failing daily since June 5. Carries the [2026-06-12] nightly item forward with a sharper diagnosis (was "root cause unknown"). -- **Surface**: CI โ€” `.github/workflows/` nightly workflow install/setup steps (`setup-uv` / `setup-node` / cache action) -- **Effort ร— Impact**: L ร— H -- **Subtracts**: no โ€” hygiene; the dep-bump gate -- **Status**: open โ€” **#518 (in 1.0.2) repointed the test/install paths but nightly STILL failed Jun 29โ€“30** (research/2026-07-03.md): a different stage (the ephemeral deploy/teardown per the `fix/nightly` work) is implicated. **Live note**: Jul 1โ€“3 nightly is green on `main` โ€” likely just-fixed; confirm the `develop` nightly (last develop run 2026-06-03) is covered before trusting it as the dep-bump gate. Consolidates the [2026-06-12] nightly item. Reviewed reviews/2026-07-03.md โ–ธ Proposal #9. - ### [2026-06-19] Ship the interactive context-breakdown badge (Cursor + LibreChat convergence) - **Source**: research/2026-06-19.md โ–ธ Top 5 #5 โ€” LibreChat v0.8.7-rc1 real-time context gauge + Cursor Context Usage Report (2026-06-05) + internal PR #433. **Reinforces** the [2026-06-05] "make the context-breakdown badge interactive" item with a second independent product datapoint. - **Surface**: frontend (context-breakdown badge component in `frontend/ai.client/src/app/session/`) @@ -233,14 +200,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: fewer-step tool turns (lower per-turn cost), best-in-class computer-use, ~4ร— fewer code-flaw pass-throughs, the `effort` compute-depth knob - **Status**: open โ€” verify Bedrock region availability (us-east-1 โœ“) and the 4.8 context window on the model card before flipping the pin; confirm the beta.27 Opus-4.7 thinking/`temperature` handling still applies -### [2026-05-29] Align MCP Apps capability advertisement to spec-canonical `io.modelcontextprotocol/ui` -- **Source**: research/2026-05-29.md โ–ธ Top 5 #3 โ€” SEP-1865 folded into the 2026-07-28 draft spec, PR #2791 (May 27) -- **Surface**: backend (inference-api `initialize` capability advertisement โ€” currently `experimental.ui`; the `ui_resource` SSE path) -- **Effort ร— Impact**: L-M ร— M -- **Subtracts**: yes โ€” retires our pre-standard `experimental.ui` identifier in favor of the conformant name -- **Unlocks**: RC-conformant negotiation with future MCP hosts/servers once the spec stabilizes (~2026-07-28) -- **Status**: open โ€” on our timeline before the RC stabilizes; diff the merged draft for any change to the declare-templates-ahead-of-time / tool-list prefetch shape - ### [2026-05-29] Compaction summary prompt: preserve standing/sensitive user instructions - **Source**: research/2026-05-29.md โ–ธ Top 5 #4 โ€” Claude Code v2.1.152 compaction-prompt change (~May 26) - **Surface**: backend (`TurnBasedSessionManager` summarization prompt) @@ -316,13 +275,51 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Resolved +### [2026-08-14] Resolve the stale review queue before `kaizen-review-prep` consumes it โ†’ RESOLVED โ€” done, in two passes +- **Decision**: Resolved. Executed in **two** passes on the same day: (1) commit `a49d2656` on the research PR, at Phil's request, resolved **nine** entries โ€” four `bedrock-agentcore` bumps, two Strands bumps, two nightly-CI entries, two MCP Apps spec-prep entries โ€” and carried the real residue forward (#564; the un-adopted Strands capabilities); (2) this review pass resolved the **three** it left: this entry itself, plus the two duplicate caching-audit entries below. +- **Reasoning**: research/2026-08-14 wrote this as an explicit instruction to review-prep because `kaizen-research` does not normally edit `## Resolved`. Phil pulled the work forward into the research PR instead, which is strictly better โ€” the ranking run then consumed a clean queue rather than one with four false premises. The entry's own status line said โœ… DONE while sitting in `## Open`, which is exactly the stub shape that gets re-ranked as live work next cycle. +- **Reviewed-in**: reviews/2026-08-14.md โ–ธ Week in Review + Retirement Candidates. + +### [2026-07-17] Audit multi-provider prompt caching + [2026-07-10] audit whether caching engages in `to_bedrock_config` โ†’ RESOLVED โ€” consolidated +- **Decision**: Superseded โ€” both consolidated into the open **[2026-07-24] "Multi-provider prompt-caching audit"** entry, which carries the same surface (`to_bedrock_config` cache-point injection plus the Mantle/OpenAI legs) and adds the GPT-5.6 explicit-breakpoint motivation on top. +- **Reasoning**: Three dated entries described **one** audit against **one** surface, differing only in accumulated motivation โ€” the same duplication pattern as the bump entries, just less loud. Issue [#642](https://github.com/Boise-State-Development/agentcore-public-stack/issues/642) stays **open** as the work-tracking issue. Two standing constraints survive the merge and must not be lost with the stubs: do **not** switch to `CacheConfig(strategy="auto")` expecting system-prompt caching (Strands #3144), and per reviews/2026-08-14.md โ–ธ Risks, per-section cache TTLs are now a hard `ValidationException` landmine (Strands #3758). +- **Reviewed-in**: reviews/2026-08-14.md โ–ธ Proposal #2 + Retirement Candidates. + +### [2026-07-24] + [2026-07-17] + [2026-07-10] + [2026-07-03] Bump `bedrock-agentcore` off 1.9.1 (โ†’1.17.0 / โ†’1.18.0 / โ†’1.18.1) โ†’ RESOLVED โ€” SHIPPED +- **Source**: research/2026-07-24.md ยท research/2026-07-17.md ยท research/2026-07-10.md ยท research/2026-07-03.md โ–ธ each Top 5 #1 +- **Decision**: Ship โ€” **done**, no further action. +- **Reasoning**: PR #857 bumped `bedrock-agentcore` **1.9.1 โ†’ 1.21.0** (with `strands-agents` 1.48โ†’1.51, `strands-agents-tools` 0.5.2โ†’0.8.6, `aws-opentelemetry-distro` 0.17โ†’0.19, `boto3` 1.43.9โ†’1.43.68 across 4 files), released in 1.14.1 on 2026-08-13. Verified 2026-08-14: latest upstream is 1.21.0 โ€” **zero version lag**. #482 (SSE deadlock, PR #563) and #571 (cross-process Memory event reorder, PR #572) are both **closed upstream**. All four entries chained through the same subject and repeated the now-false claim *"we're on 1.9.1, exposed today"*. +- **Residue carried forward**: **#564 is still open** โ€” see the [2026-08-14] guard item under `## Open`. +- **Reviewed in**: resolved directly at Phil's request 2026-08-14 (ahead of reviews/2026-08-14.md); evidence in research/2026-08-14.md โ–ธ Version-pin lag + Retirement candidates. + +### [2026-07-10] + [2026-07-03] Bump Strands (1.40 โ†’ 1.45 / โ†’ 1.47) โ†’ RESOLVED โ€” SHIPPED, capabilities split out +- **Source**: research/2026-07-10.md โ–ธ Top 5 #2 ยท research/2026-07-03.md โ–ธ Top 5 #2 +- **Decision**: Ship โ€” **bump done**; un-adopted capabilities re-queued as their own entry. +- **Reasoning**: the pin is now **1.51.0** (#857), well past both entries' targets; latest upstream is 1.52.0, so lag is 1 minor / 5 days. The [2026-07-10] entry's own status line already read "the bump itself SHIPPED". The genuinely unfinished half โ€” `continue_on_error`, optional hook ordering (#2559), `cache_tools_ttl`, `context_manager="auto"` โ€” is now the **[2026-08-14] "Adopt the Strands capabilities the 1.51 bump made available but did not wire"** entry under `## Open`. `Limits` remains separately queued at [2026-07-17]. +- **Reviewed in**: resolved directly at Phil's request 2026-08-14; evidence in research/2026-08-14.md โ–ธ Version-pin lag. + +### [2026-07-24] Fix the nightly `DELETE_FAILED` stuck ephemeral stack + [2026-06-19] Nightly `exit 127` โ†’ RESOLVED โ€” GREEN +- **Source**: research/2026-07-24.md โ–ธ Top 5 #5 ยท research/2026-06-19.md โ–ธ Top 5 #2 +- **Decision**: Ship โ€” **resolved by events**, no action needed. +- **Reasoning**: Nightly Build & Test has been **green 12 consecutive runs, Aug 3 โ†’ Aug 14**, and there have been **zero CI failures of any workflow in the last 12 days**. Both entries' premises โ€” a wedged `DELETE_FAILED` ephemeral stack, and a ~14-failure `exit 127` install cluster โ€” no longer reproduce. The dep-bump safety gate these entries existed to restore is functioning, and it vouched for #857. +- **Note**: neither entry records *which* fix closed it, so this is "resolved by observation" rather than "resolved by an identified commit". If Nightly regresses, re-open with a fresh diagnosis rather than reviving these. +- **Reviewed in**: resolved directly at Phil's request 2026-08-14; evidence in research/2026-08-14.md โ–ธ Internal Audit โ–ธ Activity. + +### [2026-07-24] Prep the MCP Apps host for the 2026-07-28 spec + [2026-05-29] Align MCP Apps capability advertisement โ†’ RESOLVED โ€” superseded (spec is now final) +- **Source**: research/2026-07-24.md โ–ธ Top 5 #4 ยท research/2026-05-29.md โ–ธ Top 5 #3 +- **Decision**: Defer into the successor entry โ€” superseded, **not** declined; the work is still wanted. +- **Reasoning**: both were written against a *moving RC* ("spec-final in 4 days", "before the RC stabilizes"). MCP **2026-07-28 is now the Current protocol version**, so the successor can be written against settled facts: the `initialize` handshake is gone, `server/discover` is the replacement (mandatory for servers, **optional for clients**), and MRTR/SEP-2322 + SEP-2243 are concrete follow-ons. โš ๏ธ Critically, the [2026-05-29] entry asserted `io.modelcontextprotocol/ui` is "spec-canonical" โ€” the 2026-08-14 scan **could not confirm that identifier** against the spec source and saw a changelog reference to preserving `experimental` settings. Leaving it open would have propagated an unverified premise into an implementation. +- **Superseded by**: **[2026-08-14] "Migrate the MCP Apps host off `initialize`/`serverInfo` to `server/discover`"** under `## Open`, which carries the capability-identifier verification as an explicit precondition. +- **Reviewed in**: resolved directly at Phil's request 2026-08-14; evidence in research/2026-08-14.md โ–ธ MCP ecosystem โ–ธ Spec status. + ### [2026-06-19] Bump Strands 1.40 โ†’ 1.44 + [2026-06-12] 1.43 + [2026-06-05] 1.42 + [2026-06-05] #2635 guard โ†’ RESOLVED โ€” superseded by the [2026-07-03] 1.45 keystone - **Decision**: Superseded โ€” all four consolidated into the open [2026-07-03] "Strands 1.40 โ†’ 1.45 + hook ordering" item. The #2635 count-tokens guard folds into the bump. - **Reviewed-in**: reviews/2026-07-03.md โ–ธ Proposal #2. ### [2026-06-19] Bump `bedrock-agentcore` 1.9.1 โ†’ 1.15.0 + [2026-06-12] 1.14.1 + [2026-05-22] 1.11.0 (ร—2) + [2026-05-22] #482 hand-written guard โ†’ RESOLVED โ€” superseded by the [2026-07-03] 1.17.0 bump -- **Decision**: Superseded โ€” all consolidated into the open [2026-07-03] "bedrock-agentcore 1.9.1 โ†’ 1.17.0" item. Per research/2026-07-03.md the #482 fix is now **upstream in 1.17.0** (PR #563), so the queued hand-written guard converts to "bump the pin" โ€” a library-native subtraction. +- **Decision**: Superseded โ€” all consolidated into the [2026-07-03] "bedrock-agentcore 1.9.1 โ†’ 1.17.0" item. Per research/2026-07-03.md the #482 fix is now **upstream in 1.17.0** (PR #563), so the queued hand-written guard converts to "bump the pin" โ€” a library-native subtraction. - **Reviewed-in**: reviews/2026-07-03.md โ–ธ Proposal #1. +- **Trail update 2026-08-14**: the [2026-07-03] item this pointed at is itself now resolved โ€” the whole chain **shipped** in #857 (1.9.1 โ†’ 1.21.0). See the [2026-07-24]+[2026-07-17]+[2026-07-10]+[2026-07-03] consolidated resolution at the top of this section. ### [2026-06-12] Add Claude Fable 5 to model settings (+ the [2026-06-19] WITHDRAW) โ†’ RESOLVED โ€” un-withdrawn, folded into the [2026-07-03] model-settings refresh - **Decision**: Superseded โ€” **NOT declined.** Fable 5 was revoked on Bedrock mid-June (forcing the withdrawal) and **reinstated Jul 1**. The reinstatement + Sonnet 5 GA are consolidated into the open [2026-07-03] "Model-settings refresh: reinstate Fable 5 + add Sonnet 5" item (US inference profile only; Global unstable). diff --git a/docs/kaizen/reviews/2026-08-14.md b/docs/kaizen/reviews/2026-08-14.md new file mode 100644 index 000000000..41da0bc88 --- /dev/null +++ b/docs/kaizen/reviews/2026-08-14.md @@ -0,0 +1,248 @@ +# Kaizen Review โ€” Friday, August 14, 2026 + +> Prepared 11:05am MT. Review window: **July 3 โ€“ August 14** (42 days โ€” six review cycles, not one). +> Source: research/2026-08-14.md + review-queue.md (**34 open** after hygiene; 46 before). +> Note: the queue-hygiene pass research/2026-08-14 asked for was pulled forward into the research PR (commit `a49d2656`, nine entries, at Phil's request) while this run was in progress. This pass resolved the three it left behind. Proposals below are ranked against the **clean** queue. + +## Week in Review + +This is not a week โ€” it is six. The last review ran **2026-07-03**; research ran on 07-10, 07-17 and 07-24, then skipped 07-31 and 08-07 and resumed today. In the gap the repo did not slow down: **43 PRs merged into `develop`, 151 non-merge commits, 7 releases (1.12.0 โ†’ 1.14.1), and zero CI failures in the last 12 days.** The single largest standing risk closed itself โ€” `bedrock-agentcore` went 1.9.1 โ†’ **1.21.0** in #857, taking a bump that had been the queue's #1 item for seven consecutive weeks to **zero version lag**, with #482 and #571 both closed upstream. + +The cost is visible in the queue. Four entries were still asserting *"we're on 1.9.1, exposed to #482 + #571 today"* โ€” false since Aug 11 โ€” and Phil cleared **nine** stale entries by hand this morning rather than let this ranking run consume them. Three items the 2026-07-03 review marked **Ship** (#405 docling, #399/#404 web-sources hardening, #480 Guardrails) are **all still open**, untouched, six weeks later. The loop caught real signal and then nothing consumed the decisions. + +Externally the window was unusually coherent: **four independent vendors shipped four different answers to "the prompt cache is a design constraint"** โ€” Anthropic's mid-conversation tool mutation that doesn't bust the prefix, Claude Code's fan-out staggering and cache-inheriting forks, MCP's server-advertised list TTLs, pydantic-ai's tools-hidden-until-revealed. We are ahead of all of them at *measuring* the problem (`cacheStatus`, fingerprint hashes, `partial_miss`, `wastedUsd`) and behind all of them at *escaping* it. Separately, MCP **2026-07-28 went final** and deleted the `initialize` handshake our MCP Apps host reads for the App-frame header โ€” the window's one confirmed breaking change against shipped code. + +## Friction โ€” the week's signal + +### Repeated patterns (โ‰ฅ2 occurrences) + +- **Ship-recommended items don't ship** (3 occurrences, all from one review) โ€” reviews/2026-07-03.md marked ten proposals; #6 (docling โ†’ close #405), #7 (web-sources helpers โ†’ close #399/#404) and #4 (Guardrails #480) were all **Ship**, and all three issues are still `OPEN` today. The two dependency bumps *did* land โ€” six weeks late, and only as a side effect of #857's general upgrade sweep, not as a kaizen action. + - *Hypothesis*: the review produces an agenda but there is no step that converts a โœ… into a tracked work item. Bumps eventually ship because they ride other work; issue-closing fixes have no such carrier. + - *Candidate fix*: on a โœ…, open the PR or a labelled issue **in the same session** as the decision. Anything not carried out of the review within a week goes back to the queue with an explicit revisit date rather than living in a merged doc. + +- **Telemetry pointed at a source nothing writes to** (2 occurrences, different root causes) โ€” PR #843 (Aug 5) fixed three Logs Insights widgets querying a phantom log group; #697's `cacheStatus` widget had **never** shown data and #838 copied the pattern. Then AWS flipped the Runtime unified-span destination default on **2026-07-20**, so any runtime recreated after that date writes spans to its own group. + - *Hypothesis*: an empty query result reads as "no errors / no traffic" instead of "broken query". This is a silent-failure *class*, now recurred with a second, unrelated cause. + - *Candidate fix*: every observability widget or Insights query asserts non-empty over a known-good window in CI, or carries a visible "0 results โ€” verify source" affordance. + +- **Derived execution contexts inherit permissions they were never granted** (4 occurrences, 3 ours + 1 upstream cluster) โ€” #852 (`isPublic` was listing-only, never enforced), the RBAC half-write trap (`grantedSkills` alone grants nothing), the `@`-mention agent fork, and upstream Claude Code 2.1.222โ€“2.1.228 fixing PreToolUse auto-allow hooks bypassing tool restrictions *in background agent tasks*. + - *Hypothesis*: our access checks are correct at the primary turn and under-specified at every derived entry point. + - *Candidate fix*: one targeted question, answered in code โ€” does a **scheduled run** or an `@`-mentioned agent evaluate `grantedTools`/`grantedSkills` against the *invoking user's* role, or against whatever the parent turn resolved? + +- **Users want an always-visible spend number** (2 occurrences, 3 days apart, independent) โ€” [#860](https://github.com/Boise-State-Development/agentcore-public-stack/issues/860) "real-time usage counter against monthly allowance on-screen" and [#861](https://github.com/Boise-State-Development/agentcore-public-stack/issues/861) "automate end-user initiated quota increase requests", both 2026-08-12, both after 1.14.0 shipped the quota runway. + - *Hypothesis*: the runway's rungs are **event-driven and dismissible**. We told users a wall exists without letting them see how close it is or do anything about it. + - *Candidate fix*: an ambient indicator reading the same `totalCost` aggregate `quota_session_notice` already uses (no new aggregation), plus a request-increase action. Both pydantic-ai and Claude Code surface **the cap and its reset time** in-message โ€” the piece the anchored-window spec still owes. + +### One-offs worth watching + +- **Revert `405d3526`** โ€” the public Cognito client for CLI PKCE auth was removed from #848. A deliberate scope walk-back, not a failure; the Textual TUI client survived. +- **[#798](https://github.com/Boise-State-Development/agentcore-public-stack/issues/798) โ€” two divergent `can_access_model` implementations (currently inert)** โ€” filed Jul 30, still open. This is the exact class CLAUDE.md already names ("they previously divergedโ€ฆ so a model could be *listed* by the catalog and *denied* on use"). Inert today, but it re-arms the moment either branch changes. +- **[#797](https://github.com/Boise-State-Development/agentcore-public-stack/issues/797) โ€” API keys survive account deprovisioning, no revocation path.** Open since Jul 30. Not surfaced by any research scan; found by internal audit only. +- **Uncommitted infra changes flagged by research this morning โ€” now resolved.** The three loose IAM edits research warned were "one `git checkout` from being lost" landed in **#870** at 14:16 today. Working tree is clean apart from a scratch file. No action needed; noted so the risk isn't re-raised next week. + +### Silence that matters + +- **Zero comments on the last two kaizen PRs.** [#719](https://github.com/Boise-State-Development/agentcore-public-stack/pull/719) (research, Jul 24) and [#533](https://github.com/Boise-State-Development/agentcore-public-stack/pull/533) (review, Jul 3) both have **no comments and no reviews**. The skill's one-week POC feedback loop โ€” Phil reviews Friday, POCs over the weekend, findings surface as PR comments the next Friday โ€” produced **nothing this cycle**. Every proposal below is therefore untested. That is the single biggest quality gap in this document. +- **`.claude/skills/angualar-best-practices/` and `.claude/skills/frontend-design/`** โ€” neither touched since **2026-04-27** (3.5 months). *Correction to research/2026-08-14.md*: it dated these to 2025-12-30 and 2026-01-18; `git log` puts the last commit touching both at 2026-04-27. Still stale, less dramatically so. +- **Nightly Build & Test: green Aug 3 โ†’ Aug 14, 12 for 12.** The dep-bump safety gate the queue has been worried about since June is live again, and it did its job โ€” #857 shipped under it. +- **`duration_ms` tool-timing** โ€” carried in the queue since 2026-05-15, surfaced in every review since, recommended **DROP** on 2026-07-03, and still sitting in `## Open`. Seventh cycle. + +## Proposals โ€” ranked + +### 1. Probe Anthropic's mid-conversation tool-mutation beta on Bedrock + +- **Source**: research/2026-08-14.md โ–ธ Top 5 #1 | review-queue.md (open since 2026-08-14) +- **Surface area**: backend โ€” `agents/main_agent/core/model_config.py` (Bedrock request construction), `_build_filtered_tools`, the `toolConfigHash` fingerprint on `C#` rows, `GET /admin/costs/sessions/{id}/calls` +- **Change**: a **probe, not a build**. Answer three questions and write them down: (a) does Bedrock Converse expose the beta at all, or is it Claude-API-only? (b) if exposed, what is the header/parameter, and does `strands` pass it through? (c) on a real session, does adding a tool mid-conversation leave `cacheReadInputTokens` intact? +- **Subtracts**: conditionally, and enormously โ€” the prompt-cache contract's most expensive clause ("any list that reaches `toolConfig` must be deterministically ordered at its source") narrows dramatically if mutation is cheap, and three separately-queued items (cross-source tool search, per-tool MCP enablement, `@`-mention prefix cost) stop being blocked by the same wall. +- **Unlocks**: per-turn tool filtering โ€” the actual fix for MCP tool-list bloat; `@`-mention switches that don't structurally re-write a 30kโ€“150k-token prefix; the Vercel app-visible-only-tools and pydantic-ai tool-search patterns become affordable. +- **Effort**: Low ยท **Impact**: High +- **POC findings**: not POCed. +- **Ship means**: run the probe against dev-ai using the existing experiment harness; record the answer in `docs/kaizen/scoping/` the way G1 and G3 were recorded. A **negative result is a valid deliverable** โ€” it closes an open assumption sitting under several queued items. +- **Decline means**: the "never mutate `toolConfig`" prohibition stays an unexamined axiom, and three queued items stay blocked on a wall we never checked was still there. +- **Recommendation**: **Ship** โ€” one day against the constraint that shapes this entire codebase, with a strong local track record of probes returning decisive answers. + +### 2. Run the Anthropic `cost_optimization` cookbook as an audit against our own contract + +- **Source**: research/2026-08-14.md โ–ธ Top 5 #4 | review-queue.md (open since 2026-08-14) +- **Surface area**: backend โ€” `core/model_config.py` (cache-point placement, `CacheConfig(strategy="auto")` at L389), `session/turn_based_session_manager.py` (truncation anchor, repeated-compaction path), `session/compaction_models.py`, plus a read of system-prompt assembly +- **Change**: work four techniques as a checklist against code we already have and **fix what it finds** โ€” no new framework. (1) **byte-stable prefix**: grep system-prompt assembly for anything time-, random-, or environment-derived (the cookbook measured a **44% swing from a single `datetime.now()`**); (2) **layered mixed-TTL breakpoints**: 54% cheaper upstream but **blocked by Strands #3758** โ€” evaluate and document why not yet, do not adopt; (3) **repeated-compaction audit**: does pass 2 preserve tool pairing without re-writing the prefix? (4) **retry cap with jitter**: an uncapped retry on a 424 or throttle re-writes the whole prefix per attempt. +- **Subtracts**: yes โ€” every item is "find waste and delete it". No new abstraction, no new dependency. +- **Effort**: Low ยท **Impact**: Mediumโ€“High +- **POC findings**: not POCed. +- **Ship means**: one PR fixing whatever items 1, 3 and 4 turn up, plus a one-paragraph note in the cost-effectiveness roadmap recording item 2's blocked status and the #3758 landmine. +- **Decline means**: we keep the instrumentation and keep not reading it โ€” which is the pattern PR #843 already caught once. +- **Recommendation**: **Ship** โ€” highest confidence-per-hour item in the scan, and item 1 doubles as a test of whether we actually read `systemPromptHash`. + +### 3. Composer draft recovery on failed send + +- **Source**: research/2026-08-14.md โ–ธ Agentic UI/UX (assistant-ui `MessageNotSentError`, 2026-08-12) | direct observation +- **Surface area**: frontend โ€” chat composer + SSE streaming state in `frontend/ai.client/src/app/` +- **Change**: hold the composer `signal()` value until the stream's first `message_start`; restore the draft on transport failure instead of losing it into an optimistic bubble that then fails. +- **Subtracts**: no โ€” addition only. **Justified**: this is the missing half of a fix we already shipped. 1.14.1's single-flight guard correctly **rejects** a duplicate turn with a 409 and still **eats the user's typed message**. The subtraction is of a user-visible failure we introduced. +- **Unlocks**: the 409-duplicate-turn path, the dropped-stream path, and the tab-switch path all stop costing the user their text. +- **Effort**: Low ยท **Impact**: Medium +- **POC findings**: not POCed. +- **Ship means**: one frontend PR, one afternoon, with a spec covering the done / abort / error paths symmetrically (assistant-ui's companion fix in the same release was exactly this asymmetry). +- **Decline means**: every rejected duplicate turn keeps destroying typed input. +- **Recommendation**: **Ship** โ€” research called it the best effort-to-user-visible-value ratio in the whole scan and it isn't in their Top 5 only because it's structurally small. It is the change Phil would notice first. + +### 4. Attack the W5 memory bill โ€” track 1 only: the LTM strategy tier + +- **Source**: research/2026-08-14.md โ–ธ Top 5 #3 | review-queue.md (open since 2026-08-14) | `project_cost_effectiveness_roadmap` (W5 is the named gap) +- **Surface area**: infrastructure / backend โ€” `infrastructure/lib/constructs/agentcore/memory-construct.ts` (the `memoryStrategies` array at L79/L85/L91, `eventExpiryDuration: 90` at L73), `apis/app_api/memory/routes.py` +- **Change**: we configure **all three built-in strategies** โ€” `semanticMemoryStrategy`, `summaryMemoryStrategy`, `userPreferenceMemoryStrategy` โ€” every one at the **$0.75/1,000 records/month** built-in tier, versus **$0.25** for override/self-managed. Measure per-strategy record volume and read frequency, then decide per strategy whether 3ร— the rate buys anything. **Explicitly deferring track 2 (Runtime Instances)** to a later cycle โ€” see Recommendation. +- **Subtracts**: yes โ€” a 3ร— per-record price cut on a line item we are paying the premium tier for **by default, with no deliberate decision behind it**. +- **Unlocks**: (track 2, deferred) Savings-Plans/ODCR-covered agent compute โ€” the first commitment-discount lever AgentCore has offered โ€” and 14-day persistent sessions, which changes the idle-reaper calculus. +- **Effort**: Medium ยท **Impact**: High +- **POC findings**: not POCed. +- **Ship means**: measure first โ€” record counts and retrieval counts per strategy from the Memory data plane, priced out against the $0.25 tier. Only then propose dropping or self-managing a strategy. +- **Decline means**: runtime memory stays ~73% of the bill and W5 stays the named open gap in the roadmap. +- **Recommendation**: **Ship track 1's measurement; Defer track 2 four weeks (revisit 2026-09-11).** Instances is the bigger prize and the weaker evidence โ€” AWS's own announcement drew **1 point and 0 comments on HN**, i.e. zero independent validation. Model it against our real session-concurrency data before it earns a slot; the $0.75โ†’$0.25 question is safe, fast, and available now. + +### 5. Audit whether derived execution contexts re-evaluate RBAC + +- **Source**: direct observation (Friction โ‰ฅ2) | research/2026-08-14.md โ–ธ Agent harness patterns +- **Surface area**: backend โ€” scheduled-runs dispatcher, the `@`-mention agent path, `apis/shared` access checks (`grantedTools` / `grantedSkills` / `effectivePermissions`) +- **Change**: answer one question in code and write the answer down: does a **scheduled run** or an **`@`-mentioned agent** evaluate `grantedTools`/`grantedSkills` against the *invoking user's* role, or against whatever the parent turn already resolved? Fix any site that inherits rather than re-evaluates. +- **Subtracts**: partial โ€” collapses four separately-discovered instances of one bug class (#852 `isPublic` listing-only, the RBAC half-write trap, the `@`-mention fork, and the scheduled-run path) into a single audited invariant instead of four hand-rolled guards. +- **Effort**: Low (audit) ยท **Impact**: Mediumโ€“High +- **POC findings**: not POCed. +- **Ship means**: a read-only audit PR documenting each derived entry point and whose role it resolves against, plus fixes for any that inherit. Close or re-scope [#798](https://github.com/Boise-State-Development/agentcore-public-stack/issues/798) in the same pass โ€” the divergent `can_access_model` pair is the same class. +- **Decline means**: we keep finding this bug one entry point at a time; it has surfaced three times in our own code in six weeks. +- **Recommendation**: **Ship** โ€” cheap, and it is the only proposal here with four independent prior occurrences. + +### 6. Ambient quota indicator + self-service increase request (#860 / #861) + +- **Source**: direct observation โ€” [#860](https://github.com/Boise-State-Development/agentcore-public-stack/issues/860) + [#861](https://github.com/Boise-State-Development/agentcore-public-stack/issues/861), both 2026-08-12 | research/2026-08-14.md โ–ธ Repeated friction +- **Surface area**: frontend (a persistent usage indicator) + backend (a request-increase action on the quota surface); reads the same `totalCost` aggregate `quota_session_notice` already uses +- **Change**: replace "we warn you at 50% and 75%, dismissibly" with "you can always see where you are, and there's a button when you get there". No new aggregation โ€” the number already exists on the session metadata row. +- **Subtracts**: partial โ€” an always-visible number makes the 50%/75% dismissible rungs partly redundant; the notice becomes the exception path rather than the only signal. +- **Unlocks**: the cap **and its reset time** surfaced in-product โ€” what both pydantic-ai and Claude Code ship, and what the drafted anchored-window cooldown spec still owes users. +- **Effort**: Medium ยท **Impact**: Mediumโ€“High +- **POC findings**: not POCed. +- **Ship means**: scope it against the drafted quota-cooldown spec so the indicator and the anchored windows agree on one model of "how much is left and when does it reset" โ€” don't ship an indicator that contradicts the spec. +- **Decline means**: two users asked in three days and got the same dismissible banner. +- **Recommendation**: **Ship** โ€” but sequence it *behind* the cooldown-windows spec decision, not in parallel. Shipping an indicator against a spend model we're about to change costs the work twice. + +### 7. Migrate the MCP Apps host off `initialize`/`serverInfo` to `server/discover` + +- **Source**: research/2026-08-14.md โ–ธ Top 5 #2 | review-queue.md (open since 2026-08-14; supersedes the [2026-07-24] and [2026-05-29] MCP Apps entries) +- **Surface area**: backend โ€” `agents/main_agent/integrations/mcp_apps.py` (the `serverInfo` capture and its consumers), `streaming/stream_coordinator.py` (`ui_resource` header emission), the `ClientCapabilities(experimental=...)` subclassing +- **Change**: resolve `serverName`/`icon` from `server/discover`, keeping the existing `manifest.json` fallback as tier two and the title-cased `ui://` authority as tier three. Keep the handshake path as a compatibility branch โ€” `server/discover` is **mandatory for servers, optional for clients**, so both eras must be served. +- **Subtracts**: yes โ€” retires the `initialize`-response dependency, and collapses the "fresh MCP session per call" concern behind the MCP Apps proxy-call 504 work, because **there is no protocol-level session left to preserve**. +- **Unlocks**: conformance with a published host matrix (Claude, VS Code Copilot, M365 Copilot, Goose, Postman); readiness for **MRTR (SEP-2322)** โ€” the sanctioned interrupt/resume shape, which would let OAuth consent and tool approval resume *without holding an SSE stream open* against the 600s timeout; readiness for SEP-2243 header-based Gateway routing. +- **Effort**: Medium ยท **Impact**: Mediumโ€“High +- **POC findings**: not POCed. +- **Ship means**: **two verifications before any code** โ€” (a) confirm the UI capability identifier against the apps **spec source**, not the docs site (this scan could not confirm `io.modelcontextprotocol/ui` vs `experimental.ui`, and coding against the wrong one is worse than waiting); (b) confirm `ui/notifications/tool-input-partial` still exists in the spec, since our `ui_tool_input_partial` relay depends on it. Then the migration PR. +- **Decline means**: App frames silently lose `serverName` and `icon` as servers upgrade โ€” **degrades quietly, never errors**. +- **Recommendation**: **Ship the two verifications this week; defer the migration until they land.** The mitigating factor is real โ€” clients may keep using the handshake โ€” so this is urgent-ish, not on fire, and building against an unconfirmed identifier is the expensive mistake here. + +### 8. Bump `docling` past 2.81.0 โ†’ close #405 + +- **Source**: review-queue.md (open since **2026-06-05**) | reviews/2026-07-03.md โ–ธ Proposal #6 (**Ship** โ€” not actioned) +- **Surface area**: backend โ€” the document-ingestion `docling` pin +- **Change**: bump off the 2.81.0 content-sniffing defect, verify a `.txt` upload succeeds, close [#405](https://github.com/Boise-State-Development/agentcore-public-stack/issues/405). +- **Subtracts**: yes โ€” a library-native bump closes a user-facing bug with no custom workaround. +- **Effort**: Low ยท **Impact**: Medium +- **POC findings**: not POCed. +- **Ship means**: bump, upload a `.txt`, close the issue. One sitting. +- **Decline means**: log it in `decisions.md` as consciously accepted โ€” `.txt` uploads have now been broken for **ten weeks** across three reviews, and pretending it's queued is worse than declining it. +- **Recommendation**: **Ship** โ€” the cleanest subtraction on the board, recommended Ship on 2026-07-03 and never picked up. If it doesn't ship this week it should be declined outright rather than carried a fourth time. + +### 9. Wire configurable Bedrock Guardrails โ€” decide in-agent vs. gateway-level in one pass (#480) + +- **Source**: review-queue.md (open since **2026-06-19**) | reviews/2026-07-03.md โ–ธ Proposal #4 (**Ship** โ€” not actioned) +- **Surface area**: backend (`BedrockModel` construction in inference-api) + infrastructure (optional `CDK_GUARDRAIL_ID` / `CDK_GUARDRAIL_VERSION` threaded to runtime env; AgentCore Policy on the Gateway construct) +- **Change**: config-wire a capability Strands already exposes (`guardrail_id`/`version`/`stream_processing_mode`/`trace`), zero-cost when unset, mirroring the `CDK_ARTIFACTS_ENABLED` optional-feature pattern โ€” **and decide in the same pass** whether one gateway-level policy is preferable to, or complements, the in-agent approach. +- **Subtracts**: partial โ€” deciding both in one pass retires the separately-queued [2026-07-03] gateway-Guardrails entry rather than running two tracks. Addition otherwise. +- **Unlocks**: deployers attach content-safety filtering and staff alerting to all model invocations without touching inference-api source โ€” the FERPA duty-of-care case for higher ed (proactive self-harm/crisis-language monitoring the reactive layer doesn't surface). A gateway-level policy is model-independent and the agent cannot reason around it. +- **Effort**: Lowโ€“Medium ยท **Impact**: High +- **POC findings**: not POCed. +- **Ship means**: verify guardrail *resource* region availability and SSE streaming-mode compatibility, then wire the env vars. Note the Mantle caveat already on record: **Mantle lacks Guardrails**, so this is a bedrock-runtime-path capability. +- **Decline means**: log it โ€” an open issue with a library-native path and a duty-of-care argument, declined twice by inaction, deserves an explicit reason. +- **Recommendation**: **Ship or Decline โ€” do not defer again.** This is the third review carrying it. It is the strongest capability-unlock on the board and it has moved zero inches in eight weeks; that gap is a decision problem, not a priority problem. + +### 10. Restore the weekly kaizen cadence, and retire two dead skills + +- **Source**: direct observation (Friction: two research Fridays and six reviews missed) | research/2026-08-14.md โ–ธ Retirement candidates +- **Surface area**: process / skills โ€” the scheduled-task config behind `kaizen-research` and `kaizen-review-prep`; `.claude/skills/angualar-best-practices/`, `.claude/skills/frontend-design/` +- **Change**: two parts. (a) Find out why 07-31 and 08-07 produced no research PR and no review ran between 07-03 and today, and fix the trigger โ€” a loop that silently stops is worse than no loop, because the queue keeps asserting stale facts (this morning it cost Phil a manual nine-entry cleanup). (b) Retire `.claude/skills/angualar-best-practices/` (untouched since 2026-04-27, **and the directory name is misspelled**, so any reference to `angular-best-practices` silently misses it, while Angular is now a full major behind) and `.claude/skills/frontend-design/` (untouched since 2026-04-27, duplicated by the `anthropic-skills:frontend-design` plugin skill now available in-session). +- **Subtracts**: yes, twice over โ€” two dead skills deleted, and a process that produced six weeks of unconsumed output either gets fixed or gets scoped down honestly. +- **Effort**: Low ยท **Impact**: Mediumโ€“High +- **POC findings**: not POCed. +- **Ship means**: check the scheduled-task history for the missed Fridays; delete the two skills (or, for the Angular one, rename and refresh it against Angular 21โ†’22 if it's still wanted โ€” a stale Angular skill is worse than none while the framework is a major behind). +- **Decline means**: accept that kaizen runs opportunistically rather than weekly, and shorten the queue's memory accordingly so it stops accumulating stale premises. +- **Recommendation**: **Ship the skill retirements; investigate the cadence.** The evidence is concrete โ€” three Ship-recommended items rotted for six weeks and four queue entries went factually false. + +> **Below the cap** (ranked 11+, carried to next review unless Phil pulls one up): +> - **Strands 1.51 primitive-duplication audit + 1.52 bump** (Lโ€“M ร— M) โ€” 1.51 (which we already run) shipped `should_offload` selective offloading and MCP name-prefix filtering that may duplicate our `toolId::name` filtering at `_build_filtered_tools` and the queued `ContextOffloader` selection logic. Possible deletion of custom code; 1.52 adds middleware-initiated interrupts on exactly the path #863 just touched. +> - **Observability non-empty assertion** (L ร— M) โ€” the durable fix for the Friction pattern above; folded into Proposal #2's spirit but distinct work. +> - **Tool-approval arg-binding HMAC** (M ร— M) โ€” Vercel's `experimental_toolApprovalSecret` binds the signature to tool name, call ID **and input arguments**, so an approved call can't be replayed with mutated args. Worth stealing outright when the queued tool-approval item is picked up. +> - **The curl version pin** (L ร— M) โ€” down-ranked, not closed. The `deb13u*` wildcard is holding (Backend Deploy green all window), so it's correctness debt rather than active breakage; the pin was never a supply-chain control. +> - **Angular 21โ†’22 / TypeScript 5.9โ†’7 lag** (H ร— M) โ€” v21 has moved to the `v21-lts` maintenance tag and TypeScript is **two majors / ~10.5 months** behind, the largest gap in the pin table. TS is downstream of the Angular decision. Needs its own scoping pass, not a proposal slot. + +## Carried Over From Prior Reviews + +- **`oauth_required` SSE flow audit** (deferred 2026-05-10 until 2026-05-24) โ€” **~12 weeks overdue, fourth surfacing.** *New context*: SEP-2322 shipped **final** as Multi Round-Trip Requests โ€” the sanctioned interrupt/resume shape, no held-open stream. *Recommendation*: **fold into Proposal #7** as its MRTR-readiness half and delete the standalone entry, or **Decline to `decisions.md`**. It must not carry a fifth review. +- **AgentCore Runtime BYO filesystem (S3 Files / EFS)** (deferred 2026-05-15 โ†’ 2026-06-12 โ†’ 2026-08-01) โ€” **now due.** *New context*: Runtime **Instances** GA brings 14-day persistent sessions; the reference repo shipped a workspace / Code-Interpreter cluster (PRs #232โ€“#234, #253) including a **storage-neutral session file API**; our own session workspace tools PR-1 is built. *Recommendation*: **keep deferred as a standalone**, and fold it into a **single agent-workspace / code-exec ADR** together with the workspace tools PR-2 scope. Revisit 2026-09-11. โš ๏ธ Three of the reference repo's six workspace commits are **security fixes** (path normalization without regex backtracking, hashed storage keys, hardened file access) โ€” if our workspace tools expose user-supplied paths, check those three holes before PR-2. +- **Named A2A agent participants in the chat UI** (deferred 2026-05-15 until 2026-06-12) โ€” precondition (an A2A *server* construct) still unmet. *New context*: `bedrock-agentcore` [#583](https://github.com/aws/bedrock-agentcore-sdk-python/issues/583) โ€” `StrandsA2AExecutor` via `serve_a2a` **never hits the AgentCore idle session timeout**, which would reintroduce the runaway-microVM class the idle reaper (#827) just fixed. *Recommendation*: **Defer** until a server construct lands, and record #583 in CLAUDE.md alongside the existing `streaming=True` guard so both traps are caught in one read. +- **`duration_ms` tool-timing into `tool_result` SSE** (carried since 2026-05-15; recommended **DROP** on 2026-07-03, never actioned) โ€” **seventh cycle.** *Recommendation*: **Decline** to `decisions.md` โ€” "deferred indefinitely; repeatedly out-prioritized, and context attribution shipped without it." + +## Retirement Candidates + +- **Twelve superseded queue entries โ€” retired today, in two passes.** Commit `a49d2656` on the research PR cleared **nine** (four `bedrock-agentcore` bumps, two Strands bumps, two nightly-CI entries, two MCP Apps spec-prep entries), carrying the real residue forward as new entries: the **#564** local guard and the un-adopted Strands capabilities. This pass cleared the **three** it left โ€” the hygiene entry itself (status said โœ… DONE while sitting in `## Open`) and the two duplicate caching-audit entries now consolidated into the open [2026-07-24] one. Queue: **46 โ†’ 34 open, with no false premises.** This is the largest single subtraction the loop has made. +- **A note on how that happened**: research/2026-08-14 wrote the hygiene as an instruction *to this skill*; Phil executed it in the research PR instead. That is strictly the better order โ€” the ranking below consumed a clean queue โ€” but it means the boundary between the two skills is fuzzier in practice than the skill files claim. Worth deciding deliberately rather than by improvisation next cycle. +- **`.claude/skills/angualar-best-practices/`** โ€” last touched 2026-04-27; misspelled directory name means `angular-best-practices` references silently miss; Angular is a full major behind. *See Proposal #10.* +- **`.claude/skills/frontend-design/`** โ€” last touched 2026-04-27; duplicated by the `anthropic-skills:frontend-design` plugin skill available in-session. *See Proposal #10.* +- **The `duration_ms` queue item** โ€” seventh cycle, never top-3, superseded in practice by shipped context attribution. *See Carried Over.* + +## Risks Acknowledged But Not Acted On + +- **MCP 2026-07-28 deleted the `initialize` handshake our MCP Apps host reads** โ€” https://modelcontextprotocol.io/specification/versioning โ€” *what breaks*: a server that upgrades stops returning `serverInfo`, so App frames silently fall back to a title-cased `ui://` authority and a generic glyph. **Degrades quietly, never errors.** Mitigating: `server/discover` is optional for clients, so servers must keep serving handshake-era clients through the transition. โ€” recommendation: **Address now** via Proposal #7's two verifications. +- **`bedrock-agentcore` #629 โ€” end-of-invocation spans dropped when the microVM freezes** โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/629 โ€” *what breaks*: cost and cache telemetry **under-reports the final turn of every session**, which is exactly where compaction and cache-write spikes land. Any EMF/trace-derived conclusion about end-of-session cost may be systematically low. โ€” recommendation: **Watch until 2026-09-11**; if unfixed, prefer the DynamoDB `C#` rows over trace data for cost conclusions and say so in the roadmap. +- **AWS flipped the Runtime span destination default on 2026-07-20** โ€” https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html โ€” *what breaks*: a runtime recreated after that date writes spans to its own log group, and a query pointed at `aws/spans` returns **empty rather than erroring** โ€” the same silent-failure class PR #843 just fixed for a different cause. โ€” recommendation: **Address now** โ€” a 20-minute check of which group our runtimes actually write to, folded into whoever touches observability next. +- **`bedrock-agentcore` #564 remains open** โ€” https://github.com/aws/bedrock-agentcore-sdk-python/issues/564 โ€” *what breaks*: a transient `ListEvents` inconsistency makes the manager treat a turn as a new session, silently losing context **and** re-writing the entire cacheable prefix. This is the one of our three tracked issues the 1.21.0 bump did **not** close. โ€” recommendation: **Address now** โ€” carried forward as its own queue entry; a local guard is cheaper than waiting on upstream. +- **Strands #3758 โ€” per-section cache TTLs emit a checkpoint order Bedrock rejects on every request** โ€” https://github.com/strands-agents/sdk-python/issues/3758 โ€” *what breaks*: nothing today (we set no per-section TTL), but it is a hard `ValidationException` landmine directly under the cookbook's 54%-cheaper layered-TTL technique. โ€” recommendation: **Accept + document** inside Proposal #2. +- **A floating model alias would silently re-point production traffic** โ€” OpenAI reaffirmed that `chat-latest` now tracks the newest ChatGPT model while **GPT-5.6 Sol remains the production API recommendation**. *What breaks*: a non-pinned identifier in the Mantle registry re-writes the cacheable prefix on whatever day OpenAI moves it โ€” the exact invisible regression `systemPromptHash`/`toolConfigHash` exist to catch. โ€” recommendation: **Address now** โ€” a one-grep audit of the Mantle model registry for any unpinned identifier. + +## What Shipped This Week + +*(42-day window โ€” 43 PRs merged, 7 releases 1.12.0 โ†’ 1.14.1. Highlights only.)* + +- **#857 โ€” `strands-agents` 1.47 โ†’ 1.51.0 and `bedrock-agentcore` 1.9.1 โ†’ 1.21.0** โ€” *the seven-week queue-topping item; takes us to zero version lag and closes #482 and #571 upstream.* +- **#863 / #864 โ€” dropped SSE streams no longer brick the next message; mid-stream disconnects are attributable** โ€” *closed the lease-leak class and made the next one diagnosable.* +- **#858 / #859 โ€” cancellation reaches in-flight MCP calls and doesn't outlive the turn; Memory writes offloaded off the asyncio event loop** โ€” *fixed the sticky-cancel bug that bricked a whole session.* +- **#827 โ€” armed AgentCore's idle reaper in `/ping`** โ€” *microVM lifetimes fell from 480โ€“520 min to 18โ€“50; the largest single cost lever shipped this window.* +- **#845 โ€” quota runway: earlier warning rungs plus a per-session cost notice** โ€” *and the direct cause of #860/#861 asking for more.* +- **#838 / #843 โ€” `partial_miss` prefix classification, and three Logs Insights widgets repointed at the real log group** โ€” *the instrument and the discovery that we weren't reading it.* +- **#852 / #808 / #807 / #796 โ€” RBAC: `isPublic` enforced, durable role-mutation audit trail, delegated admin visibility, API-key owner role resolution.** +- **#841 / #839 โ€” conversations pinned to a microVM via runtime session id; agents carrying artifact tools cached** โ€” *the G1 experiment arms; the cost thesis was disproven and a latency win found instead.* +- **#823 โ€” pre-merge guard for DynamoDB's one-GSI-per-`UpdateTable` limit** โ€” *turned a repeated deploy trap into a CI gate.* +- **#828 / #824 / #800 / #799 โ€” marketplace: live-listing updates, `taken_down โ†’ private`, version rollback, four E2E snapshot gaps.** + +## Take + +The system is trending **toward trust** and the *loop* is trending toward friction โ€” an unusual split. The code got materially better in six weeks: zero dependency lag, twelve straight green nights, the idle-reaper cost fix, three separate stream-reliability classes closed. But the kaizen loop stopped consuming its own output. Two research Fridays produced nothing, six reviews didn't happen, four queue entries went factually false, and three items this forum marked **Ship** on July 3 are still open today. That's not a signal problem โ€” the research was good and the queue caught real work. It's a conversion problem: nothing turns a โœ… into a tracked artifact, so only items that ride other people's PRs actually ship. + +The one change that matters most this week is **Proposal #1** โ€” the mid-conversation tool-mutation probe. Every other cost item on this board works *around* the prompt-cache constraint; this is the only one that asks whether the constraint still exists on Bedrock, and the answer either unblocks three queued items or closes the question for a quarter. It costs a day. If bandwidth allows exactly two, pair it with **#3 (composer draft recovery)** โ€” an afternoon, and the only proposal here a user would notice tomorrow. + +One honest caveat about this document: **every proposal is untested.** Zero comments landed on the last two kaizen PRs, so no POC findings exist to rank against. Under the skill's own bias, POC-tested items should outrank untested ones โ€” and this week there was nothing to promote. + +--- + +## Review Protocol (for Phil) + +1. Read Friction (2 min). The three-Ship-items-didn't-ship pattern is the one to react to. +2. Scan Proposals โ€” mark โœ… Ship / โŒ Decline / โธ Defer on each (4โ€“6 min). **10 proposals**; recommendation: **ship #1 (tool-mutation probe) + #3 (composer draft recovery)**, bundle **#8 (docling)** as filler, force a decision on **#9 (Guardrails)** either way, sequence **#7** behind its two verifications and **#6** behind the cooldown spec. +3. Scan Retirement Candidates โ€” same marks (1โ€“2 min). Two dead skills; the twelve queue retirements are already executed (nine by you this morning, three here). +4. Resolve Carried Over items (2 min). **Four**, three of them multiple cycles overdue โ€” `oauth_required` and `duration_ms` both need a terminal decision, not a fifth deferral. +5. Resolve the Risks block โ€” two are marked **Address now** and cost under 30 minutes each. +6. Pick 1โ€“3 to ship this week. Decline or defer the rest with a reason **and open the PR or labelled issue in the same session** โ€” that's the fix for this week's headline friction. + +Target: **12โ€“15 minutes** (longer than usual โ€” this covers six cycles, not one). + +## Post-review (for Phil โ€” separate PRs) + +- โœ… Ship items โ†’ individual feature PRs over the week. The decision is logged here; the implementation lives elsewhere. +- โŒ Decline items โ†’ appended to `docs/kaizen/decisions.md` with the reason so future research doesn't re-propose. +- โธ Defer items โ†’ kept open in `review-queue.md` with a "revisit by [date]"; surfaced again in the next review when due. diff --git a/docs/one-pagers/cost-effectiveness-roadmap.md b/docs/one-pagers/cost-effectiveness-roadmap.md index 3edcf8653..c9a3450ce 100644 --- a/docs/one-pagers/cost-effectiveness-roadmap.md +++ b/docs/one-pagers/cost-effectiveness-roadmap.md @@ -10,7 +10,9 @@ a spec, the spec wins and this page gets fixed. `agent-cache-extra-tools-bypass.md` (#834) ยท `compaction-v2-versioned-prefix.md` (#835) ยท `document-context-offload.md` + validation + evaluation (#836) ยท `quota-cooldown-windows.md` ยท `tool-search-token-bloat-strategy.md` ยท -`session-workspace-tools.md` ยท `share-large-conversations-s3-offload.md` +`session-workspace-tools.md` ยท `share-large-conversations-s3-offload.md` ยท +`agentcore-evaluations-spike-findings.md` (2026-08-12 โ€” what the managed +evaluation service does and does not do for the shared harness) *Fleet measurement:* `fleet-prefix-spend-anatomy.md` (2026-08-05) โ€” the flat, all-conversations spend decomposition this page now ranks work against. @@ -168,9 +170,25 @@ Gate summary โ€” each is a *measurement with a decision attached*, not a date: rate: *of conversations that get long enough to compact at all, 32% already exceed PR-2's proposed budget.* That rate does not shrink with growth; the population it applies to grows. -- **G3** โ€” no citations config is sent in prod today, so we don't know what - the current document path can even see. Every offload quality comparison - inherits its baseline from this probe. +- **G3 โ€” CLEARED 2026-08-12, by falsifying its own premise** + (`document-citations-probe-findings.md`). The gate existed because no + citations config is sent in prod and the #836 validation reasoned Bedrock's + visual PDF path was *tied to* citations-enabled handling โ€” which would have + made prod blind to figures. It is not. Probed with 14 questions over 5 + documents on two models: **14/14 correct in both arms, both models**, + including bar values read off an unlabeled axis, cells in a table that exists + only as pixels, and a rotated scan. **Citations turn out to be a text-layer + feature**: with citations explicitly enabled, every image-only document + returned none at all, while text-layer and mixed-document page-1 prose + questions returned them with a usable `documentPage` location. + Consequences: the offload baseline is **full visual fidelity, uncited**; the + spec's "native blocks, never flattened text" rule is now measured rather than + precautionary; and offloading an image-only document costs no citations, + because there were never any. โš ๏ธ One migration cost surfaced โ€” with citations + on, the answer text moves *inside* `citationsContent` and top-level `text` + blocks go empty, so every consumer must handle both shapes first. "Should we + enable citations?" is now a standalone product question about attribution, + **not a prerequisite for the offload arc**. Independent of all gates: #833 PR-5 (quota runway โ€” **built 2026-08-05**) and the W5 follow-ups โ€” cheap, and they don't wait on measurement. PR-5 also @@ -202,8 +220,9 @@ between sessions. **Update a row here in the PR that changes it.** | #834 spreadsheets | unbuilt | `assistant_id` into cache key + `PausedTurnSnapshot` | | #834 Memory-Space tools | unbuilt | binding descriptor into cache key | | #835 compaction v2 | unbuilt | G2 | -| #836 offload PRs 1โ€“3 | unbuilt | G3 citations probe | -| eval harness (quality veto) | **unowned** | an owner | +| #836 offload PRs 1โ€“3 | unbuilt | ~~G3 citations probe~~ โ€” **G3 cleared 2026-08-12**; baseline is full visual fidelity, uncited. Now blocked only on the eval harness owner (PRs 1โ€“3 change model-visible context) | +| G3 citations probe | โœ… **run 2026-08-12** โ€” `document-citations-probe-findings.md`; script committed at `backend/scripts/probe_document_citations.py` | nothing | +| eval harness (quality veto) | **unowned** โ€” but **smaller than the specs assumed** as of the 2026-08-12 AgentCore Evaluations spike: the managed service supplies the judges, the trajectory/tool-call scoring and the result plumbing (~a third of the build). Scope decided the same day: internal instrument, no admin feature | an owner | | replay harness (#833 ยง4.2) | partially built โ€” `experiment_agent_cache_arms.py` + `probe_runtime_session_affinity.py` drive real arms against dev; does not yet replay a recorded session's event stream | an owner for the rest | | ยง4.1 cohort scan | โœ… **run 2026-08-05** โ€” cohort is 49 sessions (1.63%) / $172.46 (21.9% of recorded session spend); D2 and D3 both reproduce outside the incident (a 174,952-char summary on another session; anchorโ‰ checkpoint on 199 of 1,238 rows). Written up in #833 ยง4.1 | nothing | | fleet spend anatomy (all conversations) | โœ… **run 2026-08-05** โ€” `fleet-prefix-spend-anatomy.md`; script committed at `backend/scripts/scan_fleet_prefix_spend.py` | nothing | @@ -221,6 +240,34 @@ between sessions. **Update a row here in the PR that changes it.** said *any* W2/W3 build PR, which #839 would have violated โ€” but #833 ยง4.3 explicitly exempts changes that alter no model-visible bytes, and #839 and #841 are both in that class. The rule was overbroad, not the merges.) + + **Scope โ€” decided 2026-08-12, previously implicit.** The harness is an + **internal instrument, not a product feature.** It exists to answer ship / + don't-ship on the four unbuilt PRs above, and then to sit idle until the next + W2/W3 change needs it. There is no admin-facing evaluation feature in this + arc, and nothing in the three specs ever proposed one โ€” the scope was simply + never written down, which is how it drifts. Concretely: no UI, no result + persistence beyond a run artifact, no per-tenant config, no RBAC surface. + Almost none of the design generalizes anyway โ€” the corpus, the + citation-page-identity canary and the pinning-boundary family are built + around document offload and compaction specifically. + + **One deliberate exception:** the *arm runner* should sit on the headless run + primitive (`agentic-platform-primitives.md` F1), not on a bespoke script. + That is the one seam a future admin-facing feature would reuse; everything + else is disposable to it. `experiment_agent_cache_arms.py` already drives + real multi-turn dev-ai sessions through the runtime gateway, so this is a + question of where the code lives, not extra work. Do **not** build corpus + generality, a config surface, or result storage now. + + **Deferred, not declined โ€” admin-facing regression checks.** Two things + changed the economics in the week before this decision: agent version + snapshots shipped (#783โ€“#801), so "did version 4 regress against version 3?" + is a question the platform can nearly ask; and the AgentCore Evaluations + spike found managed `llmAsAJudge` evaluators need **zero** infrastructure. + That makes an admin feature materially cheaper than it was โ€” size it against + the marketplace roadmap in a future planning cycle, not by folding it into + this arc, which would delay four PRs that have measured dollars behind them. - **The replay harness** (#833 ยง4.2): deterministic re-run of a session's event stream through an arm, asserting predicted vs. actual cache reads/writes per turn. Same owner as above. diff --git a/docs/specs/agentcore-evaluations-spike-findings.md b/docs/specs/agentcore-evaluations-spike-findings.md new file mode 100644 index 000000000..e39ab9ec3 --- /dev/null +++ b/docs/specs/agentcore-evaluations-spike-findings.md @@ -0,0 +1,193 @@ +# AgentCore Evaluations โ€” spike findings + +**Date:** August 12, 2026 ยท **Status:** spike complete, recommendation below +**Question:** before we write the quality-veto eval harness by hand, does the +`bedrock_agentcore.evaluation` package in our existing pin do the job? + +**Verdict: adopt it for judging and trajectory scoring; build our own arm +runner and experimental design. It removes roughly a third of the build and +cannot satisfy one requirement at all (blinding) โ€” which is worth knowing +before someone designs around it.** + +*Context:* the harness is the shared asset described by +`compaction-over-threshold-cache-spiral.md` ยง4.3, `compaction-v2-versioned-prefix.md` +ยง8, and `document-offload-evaluation.md` ยง2. All three predate the +`bedrock-agentcore` 1.21.0 bump (#857, merged 2026-08-11) and none mentions +this package. It has in fact been in the SDK since at least 1.9.1. + +--- + +## 1. What was verified, and how + +Everything below was run against **dev-ai** (`us-west-2`), not read from docs. + +| claim | evidence | +|---|---| +| The APIs exist on our pinned boto3 | boto3/botocore **1.43.68** expose `Evaluate`, `StartBatchEvaluation`, and the full evaluator CRUD on `bedrock-agentcore` / `bedrock-agentcore-control` | +| The service is authorized for us โ€” not preview-gated | `list_evaluators()` returns **16 built-in evaluators**, all `ACTIVE` | +| The whole pipeline works end-to-end **today, unmodified** | Ran `EvaluationClient.run()` against a real dev conversation: 272 spans collected, 4 scored results returned, explanations quoting the actual conversation text | +| Session correlation is already deterministic | `runtime_session_id_for()` (`apis/shared/harness/runner.py:63`) is `sid-` โ€” every conversation the existing headless harness drives maps to its spans with no new plumbing | + +### The built-in evaluators + +- **TRACE, response quality (8):** Correctness, Faithfulness, Helpfulness, + ResponseRelevance, Conciseness, Coherence, InstructionFollowing, Refusal +- **TRACE, safety (2):** Harmfulness, Stereotyping +- **SESSION (4):** GoalSuccessRate, TrajectoryExactOrderMatch, + TrajectoryInOrderMatch, TrajectoryAnyOrderMatch +- **TOOL_CALL (2):** ToolSelectionAccuracy, ToolParameterAccuracy + +Custom evaluators come in two shapes, per the `CreateEvaluator` input model: +`llmAsAJudge` (instructions + rating scale + model config โ€” fully managed, no +infra) and `codeBased` (a Lambda ARN, wrapped by the SDK's +`@custom_code_based_evaluator()` decorator). + +### Where the conversation content actually lives โ€” read this before extending anything + +This is the part a documentation-only read gets wrong, and it nearly derailed +this spike. + +`aws/spans` in dev-ai holds 117 MB of spans, and **not one of them carries +message content.** The `strands.telemetry.tracer` spans there have token counts +and model ids and nothing else; a query for any span with a non-empty `events` +array across seven days returns **zero rows**. Read only that, and the obvious +conclusion is "content capture is off, this needs observability work first." + +That conclusion is wrong. Content arrives as **OTLP log records**, not span +events โ€” emitted by `strands.telemetry.tracer` into the *runtime* log group with +`eventName = "strands.telemetry.tracer"` and a `body` of: + +``` +{"input": {"messages": [{"role": "system"|"user"|"tool", "content": โ€ฆ}, โ€ฆ]}, + "output": {"messages": [{"role": "assistant", "content": โ€ฆ}]}} +``` + +They carry `traceId` / `spanId` / `scope`, so they duck-type past the +collector's `_is_valid_adot_document` check and get swept up alongside real +spans. In the session sampled: 21 content-bearing log records among 359 +documents. `CloudWatchAgentSpanCollector` queries **both** `aws/spans` and the +runtime log group and unions them, which is why the end-to-end call works โ€” +the content comes entirely from the second query. + +Two consequences worth writing down: + +1. **Don't "fix" content capture.** It isn't broken. Setting + `OTEL_SEMCONV_STABILITY_OPT_IN` tokens to chase span events would change the + emission shape out from under a path that already works. +2. The runtime log group name is the AWS-suffixed one + (`โ€ฆ_agentcore_runtime-Z6D3HsHKs6-DEFAULT`), not the prefix-derived name. + Querying the wrong one returns zero rows rather than an error โ€” the same + trap that hid the broken dashboard widgets fixed in #843. + +### What a real result looks like + +`Builtin.Correctness` on a live dev conversation returned `value: 1.0`, label +`"Perfectly Correct"`, `tokenUsage` of 1,728 tokens, and a paragraph of +explanation that correctly identified that the agent's tool calls had 502'd, +that it declined to fabricate results, and that it had accurately reported the +failure. That is a usable judge, not a rubber stamp. + +--- + +## 2. What it does not give us + +The harness specs ask for a specific experimental design. The SDK gives a +runner, not that design. + +- **Paired A/B arms.** `OnDemandEvaluationDatasetRunner` executes one dataset + against one invoker. Arms, k=3 replicates at production temperature, + order-swapped pairwise judging, per-family reporting, McNemar โ€” all ours. +- **Blinding โ€” and this one is structural, not a gap.** The offload eval + (ยง2.4) requires stripping transcripts to *final answer text only* before + judging, because arm C transcripts contain `document_read` calls and + `` blocks that give the arm away. AgentCore Evaluations + works by feeding the evaluator the whole span set โ€” content-in-spans **is** + the mechanism. There is no scrubbing seam. **The blinded pairwise holistic + judge therefore cannot run through this service**; it needs our own judge + over scrubbed text. Everything else can. +- **Programmatic ground-truth scoring.** `ReferenceInputs` carries + `assertions` / `expected_response` / `expected_trajectory`, but exact-match + scoring against planted facts, table cells, and remapped page numbers is a + local function. Routing it through a Lambda-backed `codeBased` evaluator is + infrastructure weight for something a pure function does better. +- **Cost and token metrics.** We already read these from the `C#` rows, and + `partial_miss` (#838) is a better instrument than anything in the spans. +- **The corpus.** Planted-fact document generation, the BSU public scrape, the + chart-only pages that make the dual-encoding question answerable โ€” all ours, + and that was always where the actual thinking lived. + +### Operational notes + +- The on-demand runner sleeps `evaluation_delay_seconds` (**default 180s**) for + span ingestion, then the collector polls on top of that. Batch the arms; + don't put this in a tight loop. +- Each `evaluate` call is billed LLM usage (~1.7k tokens for one TRACE-level + Correctness call). At 120 tasks ร— 3 replicates ร— 2 arms this is real but + modest โ€” budget it, and prefer the free trajectory matchers where they answer + the question. +- Evaluator level drives batching: SESSION sends one request, TRACE fans out + over trace ids, TOOL_CALL over tool span ids, capped at 10 targets per + request. + +### One scoping decision to make deliberately + +The `body` payload carries the **full system prompt and every user message** +into an AWS-managed evaluator. For the synthetic corpus the harness is designed +around, that is a non-issue โ€” the corpus is ours and contains no user content. +It becomes a real decision only if someone later points this at recorded +production conversations. Decide it then, explicitly; don't let it happen as a +side effect of reusing the same script. + +--- + +## 3. Recommendation + +**Hybrid.** Concretely: + +**Use AgentCore Evaluations for** +- the tool-trajectory families โ€” `TrajectoryInOrderMatch` and + `ToolSelectionAccuracy` directly answer "did the model call `document_read` + when it needed to, with the right arguments", which is the + `document_read` health band the offload eval (ยง3.1) already wants +- `GoalSuccessRate` and `InstructionFollowing` for the long-session + constraint-retention families in #833 ยง4.3 +- a **secondary, non-blinded** quality signal โ€” useful as the second judge the + eval design calls for on a 20% sample + +**Build ourselves** +- the arm runner, extending `experiment_agent_cache_arms.py` (already drives + real multi-turn dev-ai sessions through the runtime gateway; don't rebuild + it โ€” and salt priming text per arm, per the G1 confound) +- the corpus and planted-fact generation +- programmatic lookup / citation / page-identity scoring +- the **blinded** pairwise holistic judge plus the scrubber, which the managed + path cannot do +- the statistics and the stopping rule + +**Net effect:** this removes roughly a third of the build โ€” the judging +infrastructure, tool-trajectory scoring, and result plumbing โ€” and leaves the +experimental design, which is the part that was always going to need care. It +also converts one open spec question ("who writes the judges?") into a +configuration choice. + +**What it does not change:** the harness is still a prerequisite for #833 PR-2, +#833 PR-4, #835 v2, and #836 PRs 1โ€“4, and it still needs an owner. It is now a +smaller job than the specs assumed. + +--- + +## 4. Follow-ups this spike opens + +1. **The three eval spec sections should be amended** to name the hybrid split + โ€” otherwise the next reader re-derives it, or worse, designs the blinded + judge on top of a service that cannot blind. +2. **`Builtin.Faithfulness` is worth a second look for the offload work + specifically.** It scores whether a response is supported by provided + context โ€” which is close to the exact question "did the digest lose + something the document had". Not in any spec today; may be a better primary + than a hand-rolled rubric for the holistic family. +3. **Online evaluation configs** (`CreateOnlineEvaluationConfig`) went + unexamined here. If they sample live traffic continuously, that is a + candidate for the "standing verification in prod" the spiral spec asks for + in ยง4.4 โ€” but it points a managed evaluator at real user conversations, so + read ยง2's scoping note first. diff --git a/docs/specs/bedrock-managed-kb-evaluation.md b/docs/specs/bedrock-managed-kb-evaluation.md index 1f5e45b05..1eea2b3ab 100644 --- a/docs/specs/bedrock-managed-kb-evaluation.md +++ b/docs/specs/bedrock-managed-kb-evaluation.md @@ -1,6 +1,7 @@ # Bedrock Managed Knowledge Base โ€” evaluation and target topology -**Status:** Evaluation complete, design proposed. No branch, no code. +**Status:** Evaluation complete, target topology proposed. Benchmark and +implementation-readiness gates defined below; no product implementation. **Question asked:** Can Bedrock Managed Knowledge Base replace our custom RAG pipeline? Given usage, pricing and quotas, should a user have *multiple* KBs per agent, or one KB filtered by agent id? And is it a fit for impromptu document @@ -27,7 +28,7 @@ conversation attachments), `docs/specs/RAG_KEEP_WARM_SPEC.md` | Embed | `backend/src/apis/shared/embeddings/bedrock_embeddings.py:24` | `amazon.titan-embed-text-v2:0`, 1024-dim, hardcoded in Python; CDK carries a *separate* `config.ragIngestion.embeddingModel` used only for the IAM ARN | | Vector store | `infrastructure/lib/constructs/rag/rag-data-construct.ts:97` | **Amazon S3 Vectors**, raw `CfnResource`. **One global index for the whole deployment**; isolation is a metadata filter only | | Retrieval | `backend/src/apis/shared/assistants/rag_service.py:18` | Pre-turn prompt augmentation, `top_k=5`, `filter={"assistant_id": ...}`. No agentic retrieval, **no reranking**, no hybrid search | -| Context cap | `rag_service.py:139`, `max_context_length=2000` | **~500 tokens reach the model** regardless of what was retrieved | +| Context cap | `rag_service.py:139`, `max_context_length=2000` | **2,000 characters (~500 tokens) reach the model** regardless of what was retrieved | | Doc-status post-filter | `rag_service.py:71` | Up to 5 *serial* DynamoDB `get_item` calls on the critical path of every RAG turn | | Re-index | `backend/src/apis/app_api/kb_sync/` + `infrastructure/lib/constructs/kb-sync/` | Shipped. Re-stages bytes to the **same S3 key** to re-fire the pipeline | | Ingestion Lambda | ARM64, 3008 MB, 900 s, `backend/Dockerfile.rag-ingestion` | **~175 s cold / ~35 s warm**; ~140 s of that is Docling/PyTorch model load. The keep-warm rule in `RAG_KEEP_WARM_SPEC.md` was never implemented | @@ -117,14 +118,12 @@ win dominates; above that it needs a deliberate decision. Two things the earlier draft got wrong, both found by calling the real API. -### 4.1 The SDK pin predates the feature +### 4.1 The SDK prerequisite is now satisfied -`boto3 1.43.9` (pinned; released **2026-05-15**) predates Managed KB GA -(**2026-06-17**) and has no `MANAGED` enum โ€” -`knowledgeBaseConfiguration.type` offers only `VECTOR | KENDRA | SQL`. -`bedrock-agentcore-control` has no KB operations at all. - -For evaluation, side-load the newer service model rather than moving the pin: +At evaluation time the repo pinned `boto3 1.43.9` (released **2026-05-15**), +which predates Managed KB GA (**2026-06-17**) and has no `MANAGED` enum โ€” +`knowledgeBaseConfiguration.type` offers only `VECTOR | KENDRA | SQL`. The +probe therefore side-loaded the newer service model without changing the repo: ``` curl -fsSL https://raw.githubusercontent.com/boto/botocore/1.43.68/botocore/data/bedrock-agent/2023-06-05/service-2.json \ @@ -132,8 +131,14 @@ curl -fsSL https://raw.githubusercontent.com/boto/botocore/1.43.68/botocore/data AWS_DATA_PATH=$MODELS python ... ``` -Implementation requires an actual `boto3`/`botocore` bump โ€” an exact-pin change, -and the first hard prerequisite of any PR-1. +The current repo now pins **`boto3==1.43.68`**, the same model version used by +the probe. The dependency bump is no longer an implementation prerequisite. +Before a benchmark or PR-1, run the create/ingest/retrieve smoke probe using the +normal checked-in environment with no `AWS_DATA_PATH`; that is the contract test +that the lock and packaged service model are really sufficient. + +`bedrock-agentcore-control` still has no KB operations. Provisioning and direct +ingestion use `bedrock-agent`; retrieval uses the Bedrock agent runtime client. ### 4.2 A MANAGED KB rejects the classic data-source types @@ -365,13 +370,20 @@ aws ce get-cost-and-usage --profile dev-ai --region us-east-1 \ ## 9. Migration surface -**Displaced:** `RagIngestionLambdaConstruct`; the `AWS::S3Vectors::*` resources in -`RagDataConstruct`; `backend/Dockerfile.rag-ingestion` (~1.5 GB of baked -Docling/PyTorch); `apis/app_api/documents/ingestion/**`; +**Target-state displaced:** `RagIngestionLambdaConstruct`; the +`AWS::S3Vectors::*` resources in `RagDataConstruct`; +`backend/Dockerfile.rag-ingestion` (~1.5 GB of baked Docling/PyTorch); +`apis/app_api/documents/ingestion/**`; `apis/shared/embeddings/bedrock_embeddings.py`; `apis/shared/assistants/rag_service.py`; the two `search_assistant_knowledgebase_with_formatting` call sites -(`inference_api/chat/routes.py:1601`, `app_api/assistants/routes.py:524`). +(`inference_api/chat/routes.py:1620`, `app_api/assistants/routes.py:524`). + +None of these resources may be removed when the managed backend first ships. +They remain required for legacy writes, dual reads, migration, and rollback. +Removal is a separate final phase after every legacy KB has migrated, the +retention window has expired, and managed traffic has completed a no-rollback +observation period. **Not displaced:** `DOC#` records and provenance fields; the documents bucket; the tabular bypass (`list_spreadsheets`/`analyze_spreadsheet`, which reads S3 @@ -387,11 +399,12 @@ direct ingest. **One-way doors:** embedding type is immutable after `CreateKnowledgeBase`. -**Confounder for any A/B:** hold `max_context_length=2000` constant. Today only -~500 tokens reach the model; raise it at the same time as switching and the -managed reranker gets credit for "we finally sent more than 500 tokens". **Test -that cap on the current pipeline first** โ€” it may be the cheapest quality win -available and it costs nothing to try. +**Confounder for any A/B:** hold the 2,000-character +`max_context_length=2000` cap constant. Today only ~500 tokens reach the model; +raise it at the same time as switching and the managed reranker gets credit for +"we finally sent more than 500 tokens". **Test that cap on the current pipeline +first** โ€” it may be the cheapest quality win available and it costs nothing to +try. --- @@ -404,7 +417,7 @@ across without downtime or a perceived quality change. ### 10.1 The seam There are exactly **two** retrieval call sites -(`inference_api/chat/routes.py:1601`, `app_api/assistants/routes.py:524`), both +(`inference_api/chat/routes.py:1620`, `app_api/assistants/routes.py:524`), both through `search_assistant_knowledgebase_with_formatting`. That is the strangler seam. Introduce a backend protocol in `apis/shared/assistants/`: @@ -434,7 +447,7 @@ ships **separately and later**, or the two effects are unattributable. |---|---| | **Score direction** | โš ๏ธ Managed returns **relevance** (higher = better; measured `1.0` on an exact hit). S3 Vectors returns cosine **distance** (lower = better). Canonicalize on relevance and convert in the `S3VectorsBackend` adapter. Get this wrong and ranking silently inverts โ€” no error, just bad answers | | `top_k` | 5 on both | -| Context cap | `max_context_length=2000` on both, unchanged | +| Context cap | `max_context_length=2000` characters on both, unchanged | | Doc-status filter | Keep the `status == "complete"` post-filter on both during parity, even though managed makes it redundant (ยง10.3) โ€” removing it in the same change confounds the comparison | | Citations | Built from the same `context_chunks`; excerpt clip stays 500 chars | @@ -476,10 +489,11 @@ assistant โ‰ˆ **4 min**; 100 docs โ‰ˆ **9.5 min**. Background work, never intera **Fleet throughput ceiling:** concurrent `Ingest`+`Delete KnowledgeBaseDocuments` is **10 per account**. At ~5 s each that is ~2 docs/sec fleet-wide โ€” ~85 min for -10,000 documents. Batching helps: the user guide says **25 documents per -`IngestKnowledgeBaseDocuments` call** while a third-party report claims the API -reference caps the array at 10. **Verify the real batch limit before sizing the -migrator**; do not assume 25. +10,000 documents. Batching helps, but AWS's own documentation currently +disagrees: the user guide says **25 documents per +`IngestKnowledgeBaseDocuments` call**, while the API reference declares a +maximum array size of **10**. Treat 10 as the safe limit and probe the real API +before sizing the migrator; do not assume 25. Reuse the kb-sync topology: sparse GSI on migration state โ†’ EventBridge โ†’ dispatcher โ†’ worker, with a bounded per-tick dispatch so a bug caps out. @@ -508,7 +522,7 @@ word "vector" never appears; and the upgrade is reversible. | State | Surface | |---|---| | legacy, no action | **nothing** โ€” no badge, no nag. A KB that works needs no UI | -| upgrade available | Inline card on the KB page: what improves (better search quality, image/table understanding), how long it takes, "your knowledge base keeps working during the upgrade" | +| upgrade available | Inline card on the KB page: only improvements proven by the ยง13 benchmark, how long it takes, and "your knowledge base keeps working during the upgrade". Do not promise better image/table understanding before the corpus comparison proves it | | `shadow`/`verify` | Non-blocking progress ("Upgrading โ€” 12 of 40 documents"), KB fully usable, safe to navigate away | | `promoted` | One-time dismissible success note; no permanent badge | | failed | Plain-language reason + Retry; **stays on legacy**, which keeps working. Never a dead end | @@ -568,3 +582,211 @@ Live in dev-ai until torn down: KBs `kb-probe-empty-1`/`VZKNLS9T1F`, `kb-probe-empty-2`/`0EKHSBWBOA` (zero data sources โ€” the empty control), `kb-probe-loaded`/`DAK4HL3JU7`; IAM role `kb-billing-probe-role`. Keep until the ยง11 question 2 CE read, then delete all four. + +--- + +## 13. Required pre-build benchmark โ€” current vs managed, 1:1 + +The three small API probes in ยง5 answer service-shape questions, not whether a +replacement improves this product. Before building a product vertical slice, +run one disposable comparison harness against **the current pipeline and a +temporary Managed KB with the same documents, questions, model, and context +cap**. This is a decision gate, not production code. + +### 13.1 Minimal scope + +One script with two adapters is enough: + +| Adapter | Path | +|---|---| +| `current` | Create a clearly named temporary assistant in dev through the existing service layer โ†’ create normal `DOC#` rows โ†’ PUT to the existing documents bucket โ†’ let the existing S3-event/Docling/Titan pipeline run โ†’ poll `DOC#` โ†’ query S3 Vectors โ†’ delete the temporary assistant and its documents | +| `managed` | Create a temporary MANAGED KB using `kb-billing-probe-role` โ†’ create one `CUSTOM` connector โ†’ direct-ingest the same S3 objects โ†’ poll document state and then a canary `Retrieve` โ†’ delete the data source and KB | + +The current adapter creates **test data only** in the dev table; it changes no +schema or configuration and cleans up afterward. The managed adapter writes no +product DynamoDB data. Both use a run id in every resource name, local result +files, and an explicit cleanup command so an interrupted process is recoverable. + +Start with exactly three controlled documents: + +1. a small plain-text file; +2. a native layout-heavy PDF with columns/tables/charts; +3. an image-only scanned PDF that requires OCR. + +Put a unique canary fact in each document and define three questions with known +answers per file. Keep this small until Managed KB clears the decision gate. + +### 13.2 Measurements + +For each backend and document, record raw timestamps for: + +- upload/direct-ingest start โ†’ API accepted; +- start โ†’ pipeline reports complete/indexed; +- start โ†’ the canary is actually returned by retrieval; +- first retrieval latency and 10 immediate retrievals (p50/p95); +- expected `document_id` present in top 5, its rank, and whether the retrieved + text contains the expected answer; +- the retrieved chunks themselves for human inspection. + +`INDEXED` and retrievable are separate timestamps. Use a fresh KB for each +first-document comparison, then ingest the remaining documents into a warm KB. +This separates one-time KB warm-up from parser/OCR cost. Run at least five +samples per document class before treating a p50/p95 as meaningful. + +Add one user-level comparison after raw retrieval: send both result sets through +the same answer model, system prompt, `top_k=5`, and **2,000-character** context +cap. Raising the cap, enabling agentic retrieval, or changing the model is a +separate experiment. + +The report is one CSV/Markdown table: + +| Backend | File | Complete/indexed | Retrievable | Retrieve p50/p95 | Top-5 hit | Expected answer | +|---|---|---|---|---|---|---| + +### 13.3 Idle follow-up, not a scheduler + +Do not build a 48-hour harness first. Leave one probe KB alive after the main +run, record its id locally, and rerun retrieval plus one tiny follow-up ingest +the next morning. If that shows a cold penalty, only then expand to controlled +1 h / 6 h / 24 h / 48 h probes using separate KBs so one check cannot warm the +next. + +### 13.4 Decision gate + +Proceed to a product vertical slice only if the comparison shows: + +- no answer-quality regression on plain text; +- a measurable benefit on layout-heavy or OCR documents, or another quality + gain large enough to justify the storage premium; +- acceptable first-document delay when treated as background work; +- subsequent-ingest improvement over the current warm path; +- acceptable added retrieval latency at p95. + +If Managed KB does not clear this gate, keep S3 Vectors and test the current +2,000-character cap independently before taking on a migration. + +--- + +## 14. Implementation-readiness gates + +The topology decision is approved for evaluation, not implementation. The +following details must be written into this spec or a linked design before +PR-1. They are blocking because each one otherwise creates a leak, lockout, or +irreversible rollout failure. + +### 14.1 Durable ingestion control plane + +The browser currently creates an `uploading` `DOC#` row, receives a presigned +S3 PUT, and relies on the bucket's `ObjectCreated` notification. There is no +upload-complete API call. Managed direct ingestion therefore still needs a +durable S3-event consumer (a much smaller replacement Lambda is the simplest +shape) that: + +1. resolves the document's logical KB and engine; +2. conditionally provisions/polls the Managed KB and `CUSTOM` data source; +3. calls `IngestKnowledgeBaseDocuments` with a stable client token; +4. polls the document until indexed and actually retrievable; +5. updates `DOC#` to complete/failed with bounded retries and a durable retry + anchor. + +Do not move this work into an app-process `asyncio.ensure_future` task. During +coexistence the same consumer must route legacy documents to the existing +pipeline and managed documents to direct ingest without double-indexing them. + +### 14.2 Stable KB identity and concrete data model + +Bindings reference a stable **application `kbId`**, never the replaceable AWS +`knowledgeBaseId`. Dormancy/rehydration can create a new AWS id without changing +an Agent binding. Define exact keys, GSIs, conditional transitions, and API +models for at least: + +- `kbId`, owner and ACL/visibility; +- `retrievalEngine`, lifecycle/provisioning state, AWS KB id and data-source id; +- embedding/parser configuration (immutable choices included); +- source-byte/storage accounting and `lastRetrievedAt`; +- migration generation, progress, lease, error and rollback timestamps; +- pin/retention/listing exemptions and delete tombstones. + +Documents and sync policies are currently children of `AST#{assistant_id}`. +Before 0..N bindings, decide whether they move under `KB#{kbId}` or how a KB +shared by multiple agents owns them. A compatible phase-1 option is +`kbId == assistantId`, an absent KB record meaning virtual legacy S3 Vectors, +and promotion changing the KB record's engine while the binding ref stays put. + +### 14.3 Authorization and publication semantics + +A shareable KB needs design-time and invocation-time access checks comparable +to Memory Spaces. Define owner/editor/viewer behavior, whether an Agent grants +invoke-through access to its KB, and whether one inaccessible KB blocks the +whole turn. The runtime must resolve access for the invoking user before +retrieval. + +Marketplace versions freeze a KB ref, not its changing contents. Decide whether +published agents pin a corpus revision, require re-review after KB changes, or +may bind only immutable/publisher-managed KBs. Exemption from lifecycle cleanup +alone does not close this review bypass. + +### 14.4 Provisioning and deletion sagas + +Create the DDB KB record first in `provisioning`, then call AWS with an +idempotency token and conditionally attach the returned ids. This prevents two +simultaneous first uploads from creating two KBs and leaves a retry anchor if +the process dies. + +Use durable tombstones for whole-KB, data-source, and individual-document +deletes. Do not erase the last DDB record or let TTL remove it until AWS confirms +deletion. Keep the document-status filter during migration and make lookup +failure **fail closed**; the current legacy filter's unfiltered fallback is not +safe for deleting or access-controlled content. + +### 14.5 IAM, encryption, audit, and teardown + +Define a dedicated Bedrock KB service role with `aws:SourceAccount` and +`aws:SourceArn` confused-deputy guards, least-privilege S3/KMS access, and a +caller `iam:PassRole` grant constrained by `iam:PassedToService`. Separately +scope provisioner/migrator CRUD, direct-ingestion, and inference +`bedrock:Retrieve` permissions. Synchronous boto3 calls from async request paths +must run through `asyncio.to_thread` or an async client. + +Managed KBs are runtime-created and are not CloudFormation children. +`scripts/teardown/destroy.sh` must list and delete only resources tagged for the +project/environment **before** deleting their service role and PlatformStack. +The daily reconciler is still required for ordinary crash orphans. + +### 14.6 Enforceable cost and quota controls + +The existing 1 GB user-files precedent is not a safe default: at the managed +rate, 1 GB for each of 30,000 users is a $150,000/month exposure. Define a lower +role-tier default, per-KB and per-owner byte caps, account-wide budget and KB +count alarms, and an atomic reserve/commit/release flow based on S3 `HEAD` size +rather than the client-reported size. Define whether the owner or invoker pays +retrieval/reranking quota. Cost-allocation tags are delayed reporting, not a +real-time enforcement mechanism; owner tags must be opaque, never email/PII. + +### 14.7 Additive deployment choreography + +Ship in explicit, reversible phases: + +1. additive schema, service role, IAM, worker resources and cleanup support; +2. dual backends dark, with mixed-version compatibility; +3. ยง13 benchmark and opted-in dual-read pilot, serving legacy; +4. opt-in migration and rollback observation; +5. managed default for new KBs; +6. stop legacy writes after the fleet is migrated; +7. reclaim legacy vectors after the retention window; +8. remove the old Lambda, image/workflow jobs, S3 Vectors resources, env vars, + IAM grants and tests in a final target-state cleanup. + +Backend code must never deploy before the IAM/resources it requires, and +Platform cleanup must never deploy before all running code has stopped using +legacy resources. + +### 14.8 Minimum test matrix + +Before promotion, cover adapter parity and score direction; managed API stubs; +create/ingest/delete idempotency; crash after AWS create but before DDB update; +DDB-only and AWS-only reconciliation; uploads/deletes during migration; +fail-closed access and document status; published-agent corpus behavior; quota +reservation races; mixed old/new deployment; teardown of tagged dynamic +resources; and CDK IAM assertions. Promotion verification uses an exact source +manifest (`document_id` + content hash/generation), not doc-count parity alone. diff --git a/docs/specs/compaction-over-threshold-cache-spiral.md b/docs/specs/compaction-over-threshold-cache-spiral.md index 96c203b0a..e3b966a61 100644 --- a/docs/specs/compaction-over-threshold-cache-spiral.md +++ b/docs/specs/compaction-over-threshold-cache-spiral.md @@ -515,6 +515,19 @@ write volume to โ‰ค (prefix โˆ’ previous cached prefix) + delta. ### 4.3 Quality gate โ€” PR-2 and PR-4 change model-visible context (veto) +> **Amended 2026-08-12 โ€” build on AgentCore Evaluations, not from scratch.** +> The harness this section describes is shared with #835 ยง8 and #836 eval ยง2, +> and a spike (`agentcore-evaluations-spike-findings.md`) found the managed +> `bedrock-agentcore` evaluation service supplies a usable third of it โ€” +> verified working end to end against dev-ai on our existing pin, with no +> infrastructure. For **this** section specifically: `Builtin.GoalSuccessRate` +> and `Builtin.InstructionFollowing` map onto the constraint-retention and +> revision-continuity families below, and the trajectory matchers score tool +> use exactly. **What must still be built by hand:** the paired arms, the k=3 +> replication, the planted-constraint corpus, and the statistics. See the +> spike's ยง3 for the split, and its ยง2 for the one requirement the managed +> path cannot meet โ€” blinding. + PR-2 replaces a 40k-token verbatim edit log with an โ‰ค8k re-summary; PR-4 freezes memory retrievals for a visit. Both are context *reductions* in exactly the workload where continuity matters most (a user iterating on one diff --git a/docs/specs/compaction-v2-versioned-prefix.md b/docs/specs/compaction-v2-versioned-prefix.md index 01512428d..2c4925b01 100644 --- a/docs/specs/compaction-v2-versioned-prefix.md +++ b/docs/specs/compaction-v2-versioned-prefix.md @@ -247,6 +247,15 @@ future compaction change ("does this mutation own its inputs?"). ## 8. Evaluation +> **Amended 2026-08-12.** The quality-veto harness referenced here is now +> partly off-the-shelf: `agentcore-evaluations-spike-findings.md` verified that +> the managed `bedrock-agentcore` evaluation service supplies the judges, +> tool-trajectory scoring and result plumbing on our existing pin, with no +> infrastructure. The paired arms, corpus and statistics remain ours, and the +> **blinded** holistic judge cannot run through the managed path at all (spike +> ยง2). This does not change what v2 must prove โ€” only how much of the +> instrument has to be written. + Reuses the spiral spec's ยง4 machinery wholesale: arm-separated cost attribution (v1-triaged vs. v2 as a new arm B4 on the same `partial_miss` instrument and replay harness) and the **quality-veto** long-session eval diff --git a/docs/specs/document-citations-probe-findings.md b/docs/specs/document-citations-probe-findings.md new file mode 100644 index 000000000..0bc7afa83 --- /dev/null +++ b/docs/specs/document-citations-probe-findings.md @@ -0,0 +1,158 @@ +# G3 โ€” the citations baseline probe, run + +**Date:** August 12, 2026 ยท **Gate:** G3 (`cost-effectiveness-roadmap.md`) +**Required by:** `document-offload-evaluation.md` ยง1 +**Script:** `backend/scripts/probe_document_citations.py` (self-contained; +builds its own corpus, no fixtures, no user content) + +**Result: G3 clears, and it clears by falsifying the premise it was built on. +Production already reads charts, image tables and scanned pages at full +fidelity, with no citations config. Citations are a *text-layer* feature, not +the switch that turns on visual understanding.** + +--- + +## 1. What the gate was for + +The #836 validation pass established that we send **no citations config in +production** โ€” `DocumentHandler.create_content_block` emits `format` / `name` / +`source.bytes` and nothing else. It then reasoned that on Bedrock the visual +(page-image) PDF path is *tied to* citations-enabled document handling, and +concluded we did not know whether production could see charts at all. + +That mattered because every offload quality comparison inherits its baseline +from arm A. If arm A were blind to figures, "the digest lost the chart" would +be unmeasurable โ€” you cannot lose what was never there. + +## 2. Method + +Five documents, fourteen questions, each asked twice โ€” once with the document +block exactly as production builds it, once with `citations: {enabled: True}` +added and **nothing else changed**. + +| document | construction | what a correct answer proves | +|---|---|---| +| `chart_only.pdf` | bar chart, image-only | values read off an axis from pixels | +| `table_in_image.pdf` | table rendered as an image | cell lookup with no text layer | +| `scanned_page.pdf` | memo text rasterized, rotated 0.7ยฐ | OCR-equivalent reading | +| `text_layer.pdf` | real PDF text object, no image | **canary** โ€” if this fails the probe is broken | +| `mixed_text_and_chart.pdf` | p1 real text layer, p2 embedded JPEG chart | the realistic production shape | + +The four image-only PDFs were verified to carry **no extractable text** +(`strings` over the raw bytes finds none of the ground-truth values), so a +correct answer cannot come from a text layer. Chart-value questions are scored +with a tolerance band because the bars carry no printed labels โ€” reading them +means estimating against the axis, and an exact-match bar would be unfair. + +Models: `us.anthropic.claude-haiku-4-5-20251001-v1:0` (the dev default) and +`us.anthropic.claude-sonnet-5`. + +## 3. Results + +**Both models, both arms: 14/14 correct.** + +| family | bare | cited | +|---|---|---| +| chart-value (2) ยท chart-compare ยท chart-structure | 4/4 | 4/4 | +| table-cell (2) ยท table-compare | 3/3 | 3/3 | +| scan-fact (2) | 2/2 | 2/2 | +| text-layer canary | 1/1 | 1/1 | +| mixed: text ยท chart (2) ยท cross-page | 4/4 | 4/4 | + +Nothing diverged between arms on correctness. The model read 631 off an +unlabeled bar, found `$401` in a table that exists only as pixels, and pulled +`PR-2291` off a rotated scan. + +### Where citations actually fired + +This is the finding with teeth. Of the 14 responses in the **cited** arm, +exactly three carried `citationsContent` โ€” and *the same three on both models*: + +| id | document | question draws on | cited | +|---|---|---|---| +| `k1` | `text_layer.pdf` | the text layer | โœ… | +| `m1` | `mixed_text_and_chart.pdf` | page-1 prose | โœ… | +| `m4` | `mixed_text_and_chart.pdf` | page-1 prose **and** page-2 figure | โœ… | +| `m2`, `m3` | `mixed_text_and_chart.pdf` | page-2 figure only | โŒ | +| `c1`โ€“`c4`, `t1`โ€“`t3`, `s1`โ€“`s2` | the three image-only PDFs | pixels only | โŒ | + +Every image-only document returned plain `text` blocks and **no citations at +all**, in both models, with citations explicitly enabled. The mixed document is +the clean demonstration: same request, same document, citations on โ€” the two +questions answerable only from the figure came back uncited, while the two that +touch the prose came back cited. + +**Citations require a text layer to cite.** They are not a visual-fidelity +switch, and enabling them does not make the model see more. Both capabilities +coexist happily in one document: page-1 prose answers arrive cited with a page +location, page-2 figure answers arrive correct and uncited. + +### The citation block's shape + +``` +citationsContent: + content: [{text: "3,400 pounds"}] <- the answer + citations: [{title: "text layer canary", + sourceContent: [{text: "...verbatim source excerpt..."}], + location: {documentPage: {documentIndex: 0, start: 1, end: 2}}}] +``` + +Two things follow. + +โš ๏ธ **With citations enabled the answer text moves *inside* `citationsContent` +and the top-level `text` blocks go empty.** Any consumer that reads only +`text` sees a blank answer. This probe hit it directly โ€” a correct answer +scored as a miss until the extractor was fixed. Anything that would consume a +citations-enabled response (the SSE content path, the eval harness scorer, +`_BEDROCK_CONTENT_BLOCK_KEYS`) has to handle both shapes before citations +could be turned on anywhere. + +โœ… `location.documentPage` gives `{documentIndex, start, end}` โ€” which is +exactly the page-identity primitive `document-offload-evaluation.md` ยง2.3 needs +for the `document_read(page_range=โ€ฆ)` remapping test. That family is buildable +against a real page number rather than an inferred one. + +## 4. What this changes + +**For the offload work (#836):** + +1. **The G3 baseline is "full visual fidelity, uncited."** Arms B and C are + measured against a model that today reads charts, image tables and scans. + The bar is higher than the spec assumed. +2. **The spec's "native blocks, never flattened text" rule is now empirically + justified**, not precautionary. A text-only digest demonstrably discards + capability the current path has โ€” we measured the capability rather than + assuming it. +3. **The citation-loss concern narrows sharply.** Offloading an *image-only* + document costs no citations, because there were never any. It is only + text-layer documents where offload trades away attribution โ€” and since the + product does not send the citations config today, that trade currently costs + nothing at all. +4. **The dual-encoding premise is confirmed but re-attributed.** PDFs do get + visual understanding. It simply is not gated on the citations config, which + is what the spec had wrong. + +**For the product, separately:** "should we enable citations?" is now a clean, +independent question about answer attribution โ€” worth asking on its own merits, +with the response-shape migration in ยง3 as its real cost. It is **not** a +prerequisite for anything in the offload arc, and the offload arc should stop +treating it as one. + +## 5. Corrections to existing documents + +- `document-context-offload.md` ยง1 / `document-context-offload-validation.md`: + the claim that the visual PDF path is tied to citations-enabled handling is + **disproven**. Visual understanding is unconditional. +- `document-offload-evaluation.md` ยง1: the required baseline experiment is + **run**; ยง2.3's note that "arm A scores here reflect the ยง1 baseline probe" + resolves to *arm A is at full fidelity*, so the citation family gates B and C + against each other, as that section anticipated. + +## 6. Loose end worth knowing + +`us.anthropic.claude-sonnet-5` rejects `temperature` outright +(`ValidationException: temperature is deprecated for this model`). Our main +chat path only sends it when a value is explicitly set, and the Nova Micro +title path is unaffected โ€” so this is **latent, not a live defect**. But an +admin pinning `temperature` on a Claude-5-family model via the inference-params +surface would 400 the turn. Worth a guard when someone is next in that code. diff --git a/docs/specs/document-context-offload-validation.md b/docs/specs/document-context-offload-validation.md index 41e23da45..0980e61b1 100644 --- a/docs/specs/document-context-offload-validation.md +++ b/docs/specs/document-context-offload-validation.md @@ -242,6 +242,18 @@ Related code findings that sharpen PR-6: processing is tied to the citations-enabled document path. The evaluation must establish the *current* fidelity baseline before scoring the digest against it (see the evaluation spec ยง2). + + > **Superseded 2026-08-12 โ€” the second half of this item is wrong.** The + > first half stands: no citations config is sent, and the + > `_BEDROCK_CONTENT_BLOCK_KEYS` reasoning does conflate response-side with + > request-side. But "full visual PDF processing is tied to the + > citations-enabled document path" is **false**, and the G3 probe + > (`document-citations-probe-findings.md`) measured it: 14/14 correct in the + > *bare* arm on two models, including unlabeled bar values, image-only table + > cells, and a rotated scan. Visual understanding is unconditional. + > Citations are a **text-layer** feature โ€” enabled explicitly, they produced + > no citations at all on image-only documents. The current fidelity baseline + > is therefore **full visual fidelity, uncited**, and it is established. 2. **The bypass spec's ยง6 isolating experiment is not clean โ€” it is nearly powerless.** 921 of 974 `create_artifact` sessions *also* have spreadsheet tools enabled, so "enable caching for `create_artifact` only" changes the diff --git a/docs/specs/document-context-offload.md b/docs/specs/document-context-offload.md index e02ec3447..1dc6cf656 100644 --- a/docs/specs/document-context-offload.md +++ b/docs/specs/document-context-offload.md @@ -457,6 +457,24 @@ emits only `format`/`name`/`source.bytes`), so enabling citations is new work in PR-1, and the evaluation's ยง1 baseline probe must establish current PDF visual fidelity before any digest comparison is scored. +> **G3 probe run 2026-08-12 โ€” `document-citations-probe-findings.md`.** The +> baseline is established: **full visual fidelity, uncited.** Today's bare +> document block reads unlabeled chart bars, image-only table cells and +> rotated scans โ€” 14/14 on two models โ€” so the ยง3 "quality tension" is real +> and this spec's *offload as native blocks, never flattened text* rule is now +> measured rather than precautionary. +> +> Two revisions follow. **Citations are a text-layer feature**, not a visual +> one: enabled explicitly, image-only documents returned none at all. So +> "citations stay enabled on it" (ยง4 lifecycle, and the reassembly note) is +> not a quality requirement for image-heavy documents โ€” there is nothing to +> preserve โ€” and **enabling citations is no longer part of PR-1**. It is a +> standalone product question about attribution, with a real migration cost: +> with citations on, the answer text moves *inside* `citationsContent` and +> top-level `text` blocks go empty, so every consumer must handle both shapes. +> Where citations *do* apply โ€” text-layer documents โ€” `location.documentPage` +> supplies the page identity the evaluation's reassembly test needs. + --- ## 7. Non-goals diff --git a/docs/specs/document-offload-evaluation.md b/docs/specs/document-offload-evaluation.md index ca56a39f2..43b9a4c9e 100644 --- a/docs/specs/document-offload-evaluation.md +++ b/docs/specs/document-offload-evaluation.md @@ -41,10 +41,51 @@ model can actually answer. Every subsequent quality comparison inherits its definition of "full fidelity" from this probe, and the offload spec's "dual-encoded" premise is unverified until it runs. +> **โœ… RUN 2026-08-12 โ€” `document-citations-probe-findings.md` +> (`backend/scripts/probe_document_citations.py`). The premise above is +> disproven; the baseline is settled.** Scaled to 14 questions over 5 +> documents (the three named here, plus a text-layer canary and a mixed +> text+figure document) on two models: **14/14 correct in both arms on both +> models**. Visual understanding is **unconditional** โ€” it is not tied to the +> citations config, and arm A sees charts, image-only table cells and scans at +> full fidelity today. +> +> **"Full fidelity" for every comparison below therefore means: reads figures, +> carries no citations.** Two knock-ons for this document. ยง2.3's citation +> family resolves the way it anticipated โ€” citations are a **text-layer** +> feature (image-only documents returned none even with the config on), so +> that family gates B against C, never against A, and on image-heavy documents +> it has nothing to measure at all. And ยง2.2's insistence on rendering PDFs +> with real layout is now load-bearing: a text-extractable-only corpus would +> compare against a strictly weaker baseline than production actually has. +> +> โš ๏ธ Implementation note for whoever writes the scorer: with citations enabled +> the answer text moves *inside* `citationsContent.content[]` and top-level +> `text` blocks go empty. The probe scored a correct answer as a miss until its +> extractor handled both shapes. + --- ## 2. Answer quality +> **Amended 2026-08-12 โ€” what to build vs. what to adopt.** This section is the +> base design for the harness shared with #833 ยง4.3 and #835 ยง8. A spike +> (`agentcore-evaluations-spike-findings.md`) verified against dev-ai that the +> managed `bedrock-agentcore` evaluation service โ€” available on our existing +> pin, authorized, no infrastructure โ€” supplies a usable third of it. Read the +> spike's ยง3 before writing a runner. The split, for this section: +> +> - **Adopt:** the tool-trajectory families (`Builtin.TrajectoryInOrderMatch`, +> `Builtin.ToolSelectionAccuracy`, `Builtin.ToolParameterAccuracy`) score the +> `document_read` health band in ยง3.1 directly and for free. Consider +> `Builtin.Faithfulness` โ€” "is the response supported by the provided +> context" is close to this spec's actual question, *did the digest lose +> something the document had* โ€” as a candidate primary for the holistic +> family, or at minimum as the second judge ยง2.4 asks for on the 20% sample. +> - **Build:** the arms, the k=3 replication, the corpus and planted facts, the +> programmatic lookup/citation scoring, the statistics โ€” and the blinded +> holistic judge, for the reason in ยง2.4. + ### 2.1 Critique of the spec's ~30-task design, and what replaces it The shape (holistic / lookup / citation) is right and maps onto the failure @@ -128,6 +169,16 @@ documents since >100k-token peaks are 4ร— more common in attachment sessions): transcripts to *final answer text only* before judging, and a scrubber must verify no digest/tool artifacts survive (grep for the digest tag and tool names; reject the sample into manual review if found). + + โš ๏ธ **This requirement rules the managed evaluation service out for the + holistic judge โ€” structurally, not as a gap to work around** (added + 2026-08-12). AgentCore Evaluations works by handing the evaluator the whole + collected span set; content-in-spans *is* the mechanism, and there is no + scrubbing seam between collection and judging. The blinded pairwise judge + must therefore be ours, over scrubbed final-answer text. Everything else in + ยง2 can go through the service. Do not design around a `ReferenceInputs` + field or a custom evaluator to recover blinding โ€” the transcript reaches the + judge regardless of what ground truth is attached. - **Consistency checks:** (1) each pair judged in both orders โ€” an order-flipped verdict is recorded as a tie; (2) a second judge model on a 20 % sample, report inter-judge agreement; (3) human (Phil or delegate) diff --git a/frontend/ai.client/package-lock.json b/frontend/ai.client/package-lock.json index e4cc37cad..6dad8e1ef 100644 --- a/frontend/ai.client/package-lock.json +++ b/frontend/ai.client/package-lock.json @@ -1,12 +1,12 @@ { "name": "ai.client", - "version": "1.14.1", + "version": "1.15.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ai.client", - "version": "1.14.1", + "version": "1.15.0", "dependencies": { "@angular/cdk": "21.2.14", "@angular/common": "21.2.17", diff --git a/frontend/ai.client/package.json b/frontend/ai.client/package.json index 70a22bad7..e68f93c2a 100644 --- a/frontend/ai.client/package.json +++ b/frontend/ai.client/package.json @@ -1,6 +1,6 @@ { "name": "ai.client", - "version": "1.14.1", + "version": "1.15.0", "scripts": { "ng": "ng", "start": "ng serve", diff --git a/frontend/ai.client/src/app/services/file-upload/file-upload.service.spec.ts b/frontend/ai.client/src/app/services/file-upload/file-upload.service.spec.ts index eb30f9ffe..b9e06351a 100644 --- a/frontend/ai.client/src/app/services/file-upload/file-upload.service.spec.ts +++ b/frontend/ai.client/src/app/services/file-upload/file-upload.service.spec.ts @@ -6,12 +6,15 @@ import { signal } from '@angular/core'; import { FileUploadService, formatBytes, - isAllowedMimeType, + isAllowedMimeType, getFileExtension, + resolveMimeType, FileTooLargeError, InvalidFileTypeError, QuotaExceededError, MAX_FILE_SIZE_BYTES, + PPTX_MAX_FILE_SIZE_BYTES, + maxFileSizeFor, ALLOWED_EXTENSIONS } from './file-upload.service'; import { ConfigService } from '../config.service'; @@ -138,8 +141,25 @@ describe('FileUploadService', () => { }); it('should throw FileTooLargeError for oversized file', () => { - const file = new File(['x'.repeat(MAX_FILE_SIZE_BYTES + 1)], 'large.pdf', { - type: 'application/pdf' + const file = new File(['x'.repeat(MAX_FILE_SIZE_BYTES + 1)], 'large.pdf', { + type: 'application/pdf' + }); + expect(() => service.validateFile(file)).toThrow(FileTooLargeError); + }); + + // A deck never goes inline to Bedrock, so the 4MB inline-document + // ceiling doesn't bound it. Corporate templates with imagery clear + // 4MB routinely โ€” capping them there is what broke `template_name`. + it('should allow a .pptx above the general cap', () => { + const file = new File(['x'.repeat(MAX_FILE_SIZE_BYTES + 1)], 'deck.pptx', { + type: 'application/vnd.openxmlformats-officedocument.presentationml.presentation' + }); + expect(() => service.validateFile(file)).not.toThrow(); + }); + + it('should still reject a .pptx above the presentation cap', () => { + const file = new File(['x'.repeat(PPTX_MAX_FILE_SIZE_BYTES + 1)], 'huge.pptx', { + type: 'application/vnd.openxmlformats-officedocument.presentationml.presentation' }); expect(() => service.validateFile(file)).toThrow(FileTooLargeError); }); @@ -153,6 +173,22 @@ describe('FileUploadService', () => { const file = new File(['content'], 'test.pdf', { type: '' }); expect(() => service.validateFile(file)).not.toThrow(); }); + + // .pptx uploads feed the PowerPoint tools (read a deck, or pass one as + // a template to create_powerpoint_presentation). The backend has always + // allowed the MIME type; this list was the only thing blocking it, and + // the create-deck tool's error text tells users to upload a template. + it('should allow .pptx by MIME type', () => { + const file = new File(['content'], 'deck.pptx', { + type: 'application/vnd.openxmlformats-officedocument.presentationml.presentation' + }); + expect(() => service.validateFile(file)).not.toThrow(); + }); + + it('should allow .pptx by extension when MIME is empty', () => { + const file = new File(['content'], 'deck.pptx', { type: '' }); + expect(() => service.validateFile(file)).not.toThrow(); + }); }); describe('clearPendingUpload', () => { @@ -429,11 +465,70 @@ describe('FileUploadService', () => { }); }); + describe('maxFileSizeFor', () => { + it('should return the general cap for ordinary files', () => { + const file = new File(['c'], 'doc.pdf', { type: 'application/pdf' }); + expect(maxFileSizeFor(file)).toBe(MAX_FILE_SIZE_BYTES); + }); + + it('should return the presentation cap by MIME type', () => { + const file = new File(['c'], 'deck.pptx', { + type: 'application/vnd.openxmlformats-officedocument.presentationml.presentation' + }); + expect(maxFileSizeFor(file)).toBe(PPTX_MAX_FILE_SIZE_BYTES); + }); + + it('should return the presentation cap by extension when MIME is empty', () => { + const file = new File(['c'], 'deck.pptx', { type: '' }); + expect(maxFileSizeFor(file)).toBe(PPTX_MAX_FILE_SIZE_BYTES); + }); + + it('should not exceed the backend presentation cap', () => { + // The backend is the enforcing side; if the UI cap were larger it + // would accept decks that presign then rejects with a 400. + expect(PPTX_MAX_FILE_SIZE_BYTES).toBe(25 * 1024 * 1024); + }); + }); + + describe('resolveMimeType', () => { + it('should keep an allowed MIME the browser reported', () => { + const file = new File(['c'], 'doc.pdf', { type: 'application/pdf' }); + expect(resolveMimeType(file)).toBe('application/pdf'); + }); + + it('should resolve .pptx from the extension when MIME is empty', () => { + // Presign would otherwise receive application/octet-stream, which + // the backend allowlist rejects โ€” and a deck stored under the wrong + // MIME is invisible to read_powerpoint_presentation, which matches + // the stored type exactly. + const file = new File(['c'], 'deck.pptx', { type: '' }); + expect(resolveMimeType(file)).toBe( + 'application/vnd.openxmlformats-officedocument.presentationml.presentation' + ); + }); + + it('should override a generic octet-stream MIME using the extension', () => { + const file = new File(['c'], 'data.xlsx', { type: 'application/octet-stream' }); + expect(resolveMimeType(file)).toBe( + 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' + ); + }); + + it('should pass through an unknown type it cannot resolve', () => { + // validateFile is the gate for rejection; this helper must not + // invent a MIME for something the allowlist never accepted. + const file = new File(['c'], 'thing.exe', { type: 'application/exe' }); + expect(resolveMimeType(file)).toBe('application/exe'); + }); + }); + describe('getAcceptedFileTypes equivalent', () => { it('should return allowed extensions', () => { expect(ALLOWED_EXTENSIONS).toContain('.pdf'); expect(ALLOWED_EXTENSIONS).toContain('.png'); expect(ALLOWED_EXTENSIONS).toContain('.docx'); + // Drives the file picker's `accept` filter in chat-input. + expect(ALLOWED_EXTENSIONS).toContain('.pptx'); expect(ALLOWED_EXTENSIONS.length).toBeGreaterThan(0); }); }); diff --git a/frontend/ai.client/src/app/services/file-upload/file-upload.service.ts b/frontend/ai.client/src/app/services/file-upload/file-upload.service.ts index fe96195af..5e1e81c2c 100644 --- a/frontend/ai.client/src/app/services/file-upload/file-upload.service.ts +++ b/frontend/ai.client/src/app/services/file-upload/file-upload.service.ts @@ -8,7 +8,14 @@ import { ConfigService } from '../config.service'; export type FileStatus = 'pending' | 'ready' | 'failed'; /** - * Allowed MIME types for file uploads (Bedrock-compliant) + * Allowed MIME types for file uploads. + * + * Must stay in sync with `ALLOWED_MIME_TYPES` in + * `backend/src/apis/shared/files/models.py` โ€” the backend is the enforcing + * side, this list drives the picker's `accept` filter and the drop/paste + * check. Not every entry is sent to Bedrock as a document block: csv/xls/xlsx + * route to the spreadsheet tools and pptx routes to the PowerPoint tools + * (Bedrock's document-format enum has no `pptx`). See `_partition_attachments`. */ export const ALLOWED_MIME_TYPES: Record = { // Documents @@ -19,6 +26,7 @@ export const ALLOWED_MIME_TYPES: Record = { 'text/csv': 'csv', 'application/vnd.ms-excel': 'xls', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': 'xlsx', + 'application/vnd.openxmlformats-officedocument.presentationml.presentation': 'pptx', 'text/markdown': 'md', // Images (Bedrock-supported) 'image/png': 'png', @@ -32,7 +40,7 @@ export const ALLOWED_MIME_TYPES: Record = { */ export const ALLOWED_EXTENSIONS = [ // Documents - '.pdf', '.docx', '.txt', '.html', '.csv', '.xls', '.xlsx', '.md', + '.pdf', '.docx', '.txt', '.html', '.csv', '.xls', '.xlsx', '.pptx', '.md', // Images '.png', '.jpg', '.jpeg', '.gif', '.webp' ]; @@ -42,6 +50,34 @@ export const ALLOWED_EXTENSIONS = [ */ export const MAX_FILE_SIZE_BYTES = 4 * 1024 * 1024; +/** + * Maximum .pptx size in bytes (25MB). + * + * Decks get a larger cap because the general limit exists to keep documents + * under what Bedrock accepts as an inline content block, and a .pptx is + * never sent inline โ€” it routes to the PowerPoint tools instead. Corporate + * templates with imagery routinely clear 4MB, which is what made the + * `template_name` workflow unusable. + * + * Must match `FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION` on the backend + * (`files/service.py`), which is the enforcing side โ€” if this is the larger + * of the two, the UI accepts a deck that presign then rejects. + */ +export const PPTX_MAX_FILE_SIZE_BYTES = 25 * 1024 * 1024; + +/** + * Resolve the size cap that applies to a given file. Use this rather than + * `MAX_FILE_SIZE_BYTES` directly at any gate, or a legitimate deck gets + * rejected by whichever check was missed. + */ +export function maxFileSizeFor(file: File): number { + const isPptx = + file.name.toLowerCase().endsWith('.pptx') || + file.type === + 'application/vnd.openxmlformats-officedocument.presentationml.presentation'; + return isPptx ? PPTX_MAX_FILE_SIZE_BYTES : MAX_FILE_SIZE_BYTES; +} + /** * Maximum files per message */ @@ -231,6 +267,48 @@ export function getFileExtension(filename: string): string { return lastDot >= 0 ? filename.slice(lastDot).toLowerCase() : ''; } +/** + * Extension โ†’ MIME map, mirroring `ALLOWED_EXTENSIONS` in + * `backend/src/apis/shared/files/models.py`. + */ +const EXTENSION_TO_MIME_TYPE: Record = { + '.pdf': 'application/pdf', + '.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', + '.txt': 'text/plain', + '.html': 'text/html', + '.csv': 'text/csv', + '.xls': 'application/vnd.ms-excel', + '.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + '.pptx': 'application/vnd.openxmlformats-officedocument.presentationml.presentation', + '.md': 'text/markdown', + '.png': 'image/png', + '.jpg': 'image/jpeg', + '.jpeg': 'image/jpeg', + '.gif': 'image/gif', + '.webp': 'image/webp', +}; + +/** + * Resolve the MIME type to send with a presign request. + * + * Browsers do not always populate `File.type` โ€” an OS with no handler + * registered for the extension reports `''`, and some file managers report + * `application/octet-stream`. `validateFile` accepts those by extension, so + * sending the browser's value verbatim would hand the backend a MIME its + * own allowlist rejects, 400-ing an upload the UI just accepted. + * + * Resolving from the extension also keeps files reachable by the tools that + * route them, which match on the stored MIME *exactly*: a deck stored as + * octet-stream is invisible to `read_powerpoint_presentation`, and a + * spreadsheet stored that way never reaches the analysis tools. + */ +export function resolveMimeType(file: File): string { + if (isAllowedMimeType(file.type)) { + return file.type; + } + return EXTENSION_TO_MIME_TYPE[getFileExtension(file.name)] ?? file.type; +} + /** * Service for managing file uploads via pre-signed URLs. * @@ -313,9 +391,10 @@ export class FileUploadService { * @throws FileUploadError if validation fails */ validateFile(file: File): void { - // Check size - if (file.size > MAX_FILE_SIZE_BYTES) { - throw new FileTooLargeError(file.size, MAX_FILE_SIZE_BYTES); + // Check size (pptx has its own, larger cap) + const sizeLimit = maxFileSizeFor(file); + if (file.size > sizeLimit) { + throw new FileTooLargeError(file.size, sizeLimit); } // Check MIME type @@ -346,7 +425,7 @@ export class FileUploadService { const presignRequest: PresignRequest = { sessionId, filename: file.name, - mimeType: file.type || 'application/octet-stream', + mimeType: resolveMimeType(file), sizeBytes: file.size, }; diff --git a/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.spec.ts b/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.spec.ts index efb77a8d5..2dc94375a 100644 --- a/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.spec.ts +++ b/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.spec.ts @@ -95,4 +95,58 @@ describe('OAuthConsentService', () => { expect(service.pending().length).toBe(1); }); }); + + describe('pre-flight consents (no interruptId)', () => { + // Emitted when an OAuth-gated MCP server refused `tools/list`, so the + // tool never registered and no turn is paused. The backend re-emits + // these every turn that rebuilds the agent, so dismissal has to stick + // locally โ€” there is no server-side breadcrumb to DELETE. + const URL_A = 'https://accounts.example/consent?a=1'; + + it('surfaces a request with no interruptId', () => { + service.requestConsent('github-oauth', URL_A, undefined, 'msg-1', 'sess-1'); + + const pending = service.pending(); + expect(pending.length).toBe(1); + expect(pending[0].providerId).toBe('github-oauth'); + expect(pending[0].interruptId).toBeUndefined(); + }); + + it('dedupes by providerId while the prompt is live', () => { + service.requestConsent('github-oauth', URL_A, undefined, 'msg-1', 'sess-1'); + service.requestConsent('github-oauth', URL_A, undefined, 'msg-2', 'sess-1'); + + expect(service.pending().length).toBe(1); + }); + + it('does not resurrect after the user dismisses it', () => { + service.requestConsent('github-oauth', URL_A, undefined, 'msg-1', 'sess-1'); + service.dismiss('github-oauth'); + expect(service.pending().length).toBe(0); + + // Next turn re-emits because consent still has not landed. + service.requestConsent('github-oauth', URL_A, undefined, 'msg-2', 'sess-1'); + expect(service.pending().length).toBe(0); + }); + + it('still surfaces an interrupt-driven prompt after a pre-flight dismissal', () => { + // A paused turn is blocking on consent โ€” the user must be able to act + // even though they dismissed the passive pre-flight nudge earlier. + service.requestConsent('github-oauth', URL_A, undefined, 'msg-1', 'sess-1'); + service.dismiss('github-oauth'); + + service.requestConsent('github-oauth', URL_A, 'i-99', 'msg-2', 'sess-1'); + expect(service.pending().length).toBe(1); + expect(service.pending()[0].interruptId).toBe('i-99'); + }); + + it('clear() lifts the dismissal', () => { + service.requestConsent('github-oauth', URL_A, undefined, 'msg-1', 'sess-1'); + service.dismiss('github-oauth'); + service.clear(); + + service.requestConsent('github-oauth', URL_A, undefined, 'msg-2', 'sess-2'); + expect(service.pending().length).toBe(1); + }); + }); }); diff --git a/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.ts b/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.ts index 4acbf333c..b04ba8805 100644 --- a/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.ts +++ b/frontend/ai.client/src/app/services/oauth-consent/oauth-consent.service.ts @@ -110,6 +110,13 @@ export class OAuthConsentService { * from `toolUseId`), so legitimate prompts are never suppressed. */ private readonly seenInterruptIds = new Set(); + /** Providers whose *pre-flight* consent prompt the user dismissed. Those + * prompts carry no interruptId and the backend re-emits them every turn + * the agent is rebuilt, so `seenInterruptIds` can't suppress them. + * Cleared by {@link clear} on session change / logout, and by a + * successful consent, so this only silences the current tab session. */ + private readonly dismissedPreflightProviders = new Set(); + /** ProviderIds whose popup is currently open. */ private readonly inFlight = signal>(new Set()); @@ -209,6 +216,15 @@ export class OAuthConsentService { if (interruptId) { this.seenInterruptIds.add(interruptId); } + // Pre-flight consents (no interruptId โ€” see OAuthRequiredEvent) are + // re-emitted on every turn that rebuilds the agent, because the tool + // keeps failing its `tools/list` pre-flight until consent lands. Honour + // an explicit dismissal so declining once doesn't mean answering the + // same prompt after every message. Interrupt-driven prompts are exempt: + // there a turn is genuinely paused and the user must be able to act. + if (!interruptId && this.dismissedPreflightProviders.has(providerId)) { + return; + } this.requests.update((map) => { const next = new Map(map); next.set(providerId, { @@ -266,6 +282,8 @@ export class OAuthConsentService { // another tab). Drop the request and let the resume handler โ€” if // any โ€” fire so the agent can finish the turn. this.dismiss(providerId); + // Connected, not declined โ€” same reasoning as handleCompletion. + this.dismissedPreflightProviders.delete(providerId); if (request.interruptId && this.resumeHandler) { void Promise.resolve( this.resumeHandler([request.interruptId], { sessionId: request.sessionId }), @@ -455,6 +473,13 @@ export class OAuthConsentService { const sessionId = entry?.sessionId; const interruptId = entry?.interruptId; + // A pre-flight prompt has no server-side breadcrumb to clear, so the + // only way to make the dismissal stick against next turn's re-emission + // is to remember it here. + if (entry && !interruptId) { + this.dismissedPreflightProviders.add(providerId); + } + this.requests.update((map) => { if (!map.has(providerId)) { return map; @@ -500,6 +525,7 @@ export class OAuthConsentService { this.blocked.set(new Set()); this.lastCompletion.set(null); this.seenInterruptIds.clear(); + this.dismissedPreflightProviders.clear(); } /** Acknowledge the last completion signal after the UI has reacted. */ @@ -526,6 +552,11 @@ export class OAuthConsentService { // interrupt server-side, so a separate DELETE would just be redundant. const request = this.requests().get(message.providerId); this.dismiss(message.providerId, { syncServer: false }); + // dismiss() treats an interruptId-less entry as a user decline and + // suppresses future pre-flight prompts. This path is the opposite โ€” + // consent succeeded โ€” so undo that: if the user later disconnects the + // connector, the prompt must be able to come back. + this.dismissedPreflightProviders.delete(message.providerId); if (!request?.interruptId || !this.resumeHandler) { return; diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts index 7c42ad64c..c967afede 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts @@ -30,7 +30,7 @@ import { FileUploadService, PendingUpload, ALLOWED_EXTENSIONS, - MAX_FILE_SIZE_BYTES, + maxFileSizeFor, MAX_FILES_PER_MESSAGE, formatBytes } from '../../../services/file-upload'; @@ -647,11 +647,12 @@ export class ChatInputComponent { // Validate and upload each file for (const file of newFiles) { - // Check file size - if (file.size > MAX_FILE_SIZE_BYTES) { + // Check file size (pptx has its own, larger cap โ€” see maxFileSizeFor) + const sizeLimit = maxFileSizeFor(file); + if (file.size > sizeLimit) { this.toastService.error( 'File Too Large', - `${file.name} exceeds maximum size of ${formatBytes(MAX_FILE_SIZE_BYTES)}.` + `${file.name} exceeds maximum size of ${formatBytes(sizeLimit)}.` ); continue; } diff --git a/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.spec.ts b/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.spec.ts new file mode 100644 index 000000000..ecc014292 --- /dev/null +++ b/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.spec.ts @@ -0,0 +1,41 @@ +import { describe, it, expect } from 'vitest'; +import { + FILE_TYPE_STYLES, + DEFAULT_STYLE, +} from './file-attachment-badge.component'; +import { ALLOWED_MIME_TYPES } from '../../../../../services/file-upload'; + +/** + * Every uploadable type needs its own card style. + * + * A type missing from FILE_TYPE_STYLES doesn't fail loudly โ€” it silently + * falls through to DEFAULT_STYLE and the card renders a generic grey "FILE" + * chip. That is exactly how .pptx shipped: the upload allowlist gained the + * type but this map didn't, so decks arrived looking like anonymous blobs. + * + * Pinning the map against the upload allowlist means the next type added to + * one has to be added to the other. + */ +describe('file attachment card styles', () => { + const PPTX_MIME = + 'application/vnd.openxmlformats-officedocument.presentationml.presentation'; + + it('gives .pptx its own style rather than the generic fallback', () => { + const style = FILE_TYPE_STYLES[PPTX_MIME]; + expect(style).toBeDefined(); + expect(style.label).toBe('PPTX'); + expect(style.label).not.toBe(DEFAULT_STYLE.label); + }); + + it('uses a presentation icon for .pptx, not a plain document', () => { + expect(FILE_TYPE_STYLES[PPTX_MIME].icon).toBe('heroPresentationChartBar'); + expect(FILE_TYPE_STYLES[PPTX_MIME].icon).not.toBe(DEFAULT_STYLE.icon); + }); + + it('styles every mime type the upload allowlist accepts', () => { + const unstyled = Object.keys(ALLOWED_MIME_TYPES).filter( + (mime) => !(mime in FILE_TYPE_STYLES), + ); + expect(unstyled).toEqual([]); + }); +}); diff --git a/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.ts b/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.ts index 603681f96..973fa3c1d 100644 --- a/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.ts +++ b/frontend/ai.client/src/app/session/components/message-list/components/file-attachment/file-attachment-badge.component.ts @@ -6,6 +6,7 @@ import { heroTableCells, heroCodeBracket, heroPhoto, + heroPresentationChartBar, heroArrowTopRightOnSquare, } from '@ng-icons/heroicons/outline'; import { MarkdownComponent } from 'ngx-markdown'; @@ -22,14 +23,14 @@ interface FileTypeStyle { header_bg: string; } -const DEFAULT_STYLE: FileTypeStyle = { +export const DEFAULT_STYLE: FileTypeStyle = { icon: 'heroDocument', label: 'FILE', accent_text: 'text-gray-600 dark:text-gray-300', header_bg: 'bg-gray-50 dark:bg-gray-700/50', }; -const FILE_TYPE_STYLES: Record = { +export const FILE_TYPE_STYLES: Record = { 'application/pdf': { icon: 'heroDocument', label: 'PDF', @@ -72,6 +73,15 @@ const FILE_TYPE_STYLES: Record = { accent_text: 'text-green-600 dark:text-green-300', header_bg: 'bg-green-50 dark:bg-green-950/40', }, + // Orange is PowerPoint's brand association, which makes the chip readable + // at a glance. It overlaps with HTML's accent, but the label text + // disambiguates and the two rarely appear in the same conversation. + 'application/vnd.openxmlformats-officedocument.presentationml.presentation': { + icon: 'heroPresentationChartBar', + label: 'PPTX', + accent_text: 'text-orange-600 dark:text-orange-300', + header_bg: 'bg-orange-50 dark:bg-orange-950/40', + }, 'text/markdown': { icon: 'heroDocumentText', label: 'MD', @@ -112,6 +122,12 @@ const THUMBNAIL_PREVIEW_MIMES = new Set(['application/pdf']); /** Skeleton "lines of text" widths (percent), tuned to look like a paragraph. */ const SKELETON_LINE_WIDTHS = [92, 78, 88, 64, 95, 70, 84, 58]; +const PRESENTATION_MIME = + 'application/vnd.openxmlformats-officedocument.presentationml.presentation'; + +/** Bullet-row widths (percent) inside the mock slide. */ +const SLIDE_BULLET_WIDTHS = [78, 92, 60]; + /** * Document-style preview card for a non-image file attachment. * @@ -133,6 +149,7 @@ const SKELETON_LINE_WIDTHS = [92, 78, 88, 64, 95, 70, 84, 58]; heroTableCells, heroCodeBracket, heroPhoto, + heroPresentationChartBar, heroArrowTopRightOnSquare, }), ], @@ -247,13 +264,49 @@ const SKELETON_LINE_WIDTHS = [92, 78, 88, 64, 95, 70, 84, 58];
- - + + @if (!isPresentation()) { + + } - @if (thumbnailUrl(); as url) { + @if (isPresentation()) { + + + } @else if (thumbnailUrl(); as url) { - @if (!thumbnailUrl()) { + so the rendered page edge stays crisp, and for decks, where the + slide is fully visible and a fade would just dim its lower edge. --> + @if (!thumbnailUrl() && !isPresentation()) {