Observability: customizable application-layer outcomes from evaluation runs

[zzkamzn]

Summary

The Observability page so far reports infrastructure signals: latency, errors, turns and cost. This PR adds customizable application-layer observability next to them: whether the agent's answers were right, or broke a business rule.

Operators define what "right" means per use case, with no code change. They create an evaluation dataset with their own scenario name and scoring contract. The page then discovers the scenario and charts its results next to the execution facts of the same calls. Adding a new scoring method is a documented code extension point.

• Scenarios and scoring per dataset.
  • An evaluation dataset has a free-form scenario name and a scoring config.
  • json_exact compares one JSON output field with the expected value, deterministically and with no judge call. The field can be dotted, such as decision.intent. Matching is strict and case-sensitive, and the docs spell out the contract.
  • llm_judge keeps the judge and accepts an optional business rubric.
• Version-pinned runs.
  • A run records the published agent's version when it starts. If the agent is republished mid-run, the run fails, so one run never mixes two versions.
  • The version's system prompt is stored once and shared by every run of that version.
• Evidence stored per case. Each case's prompt, expectation, full answer and verdict is its own DynamoDB item. Long multilingual runs stay far below the item size limit, and a run is read by id.
• Execution facts on the same record. Each case records its agent call and judge call: ok, duration, turns, cost and runtime session id. The run aggregates them, so outcome and execution health of the same cases sit side by side with no join against the invocation ledger. A dataset's run history is indexed, so the previous run is always found for comparison.
• Observability page.
  • The existing invocation ledger view is unchanged.
  • A separate "Application outcomes" section discovers scenarios from the runs. For each run it shows:
    • The same calls' infrastructure view: ok count, median latency and cost.
    • The pass rate with its sample size.
    • The value distribution, invalid outputs and low-score cases.
    • Each case's answer, verdict and call metrics.
  • It compares runs of the same dataset and flags a drop only when it amounts to at least two cases.
  • It loads independently of the ledger and starts no model calls.
• Example scenarios that exercise the feature end to end. They are synthetic, and the answers are real model calls.
  • scripts/run_observability_demo.py seeds a classifier and a support benchmark.
  • scripts/run_difficult_customer_evals.py adds the oncall, rnd and sales-success scenarios and 96 adversarial cases in two rounds:
    • hard: prompt injection, forged system messages, format overrides, social engineering, over-policy demands, several languages.
    • extreme: contradictory stakeholders, profanity, personal attacks, blame-shifting, unverifiable promises.
  • Both scripts take --prefix and refuse to republish an agent they did not create.
  • The difficult-customer script ends by printing each run's infrastructure view next to its pass rate.

What it shows on a live deployment

I ran both scripts against a us-east-1 deployment, using real Bedrock calls on the portal's published agents. The inputs are synthetic; nothing else is fabricated.

Run on the deployed version of this branch. Every number below comes from the runs themselves, as shown on the page:

Dataset	Scoring	Cases	Agent calls ok	Median latency	Passed
hard-customers-classification	json_exact	17	17	5.5 s	12
hard-customers-support	judge	12	12	6.9 s	12
oncall-triage-hard	json_exact	14	14	7.7 s	14
rnd-assistant-hard	judge	12	12	12.2 s	12
sales-cs-hard	judge	12	12	8.5 s	9
extreme-support	judge	8	8	10.7 s	7
extreme-oncall	json_exact	8	8	10.6 s	7
extreme-rnd	judge	6	6	13.8 s	5
extreme-sales	judge	7	7	13.1 s	5
total		96	96		83

All 96 agent calls returned ok, with normal latency and cost, so infrastructure dashboards would call this healthy. The evaluation layer flagged 13 answers that broke a rule. Examples:

• A forged system notice changed the classifier's output. The message announced a new category, urgent, and the classifier returned it. In the previous round the same prompt, unchanged, resisted it. A re-run of the same dataset surfacing a behaviour change is exactly what the comparison view is for.
• An upsell to a Standard customer who was demanding compensation.
• Taking a side in a PM-versus-TL argument instead of naming who decides.
• A sev3 alert talked down to sev4 by an angry message.
• Leading with the maximum discount when a customer only asked for a price.

Each failure on the page shows its answer, the verdict, the agent version, and the call's latency, cost and runtime session id.

Judge verdicts are screening signals. The judge shares a kernel and model family with the agents it grades. In this round it faulted a reply for "unverified pricing", although $45 and the 15% annual discount are in the agent's own policy. In an earlier round it passed an answer that invented a response deadline. The page keeps every full answer so a reviewer can confirm or overturn each verdict.

Test plan

• backend/tests/test_eval_scenarios.py: 13 tests pass. They cover:
  • json_exact parsing, including dotted fields.
  • Judge-verdict validation.
  • The version-change guard.
  • Per-case storage size in the worst-case CJK run.
  • Read by id.
  • One prompt snapshot per version.
  • Legacy runs remaining readable.
  • Per-case call metrics and the run's aggregate.
  • Dataset history surviving 60 runs of another dataset.
• The storage path was also exercised against moto's DynamoDB.
• tsc -b and vite build pass on this branch, which is based on current main.
• Deployed backend, frontend and the schedule-runner Lambda to a live stack and re-ran both scripts end to end.
• Reviewer: run the two scripts on a deployment and check the Application outcomes tabs and the unchanged ledger table.

Follow-ups, online evaluation of production traffic and where telemetry should live, are tracked in #59.

🤖 Generated with Claude Code

[odinwang]

Thanks for this. The core finding is the useful part: 67 calls succeeded at the infrastructure layer with normal latency and cost, and the evaluation layer still flagged 10 answers that broke a business rule. Infrastructure metrics cannot show that, and having it measured on real calls makes the case better than any write-up would. The 96 adversarial cases are a solid asset that we can reuse whatever direction evaluation takes next. Several design choices are good: deterministic json_exact for structured outputs, llm_judge with a business rubric for open answers, version pinning for comparable runs, and keeping full evidence so a reviewer can overturn a verdict. The two fixes to _parse_verdict (a string "false" no longer passes) and to skipping the judge when the agent call fails are also correct.

The items below need changes before merge. Most are about storage limits and keeping the existing page intact.

Must fix

1. Run items can exceed the DynamoDB 400 KB item limit.
_execute now stores up to 2,000 chars of prompt, 1,000 of expected and 8,000 of answer per case, plus a system prompt snapshot of up to 20,000 chars, and rewrites the whole results list after every case. With 20 cases and CJK text (3 bytes per char) a run can pass 400 KB partway through, update_item fails, and the run is marked failed. A side effect: list_runs(50) is a single Query page (1 MB max), so larger items mean fewer runs come back, and get_run scans only that page, so an existing run can return 404 while the UI is polling it.
Suggested direction: store each case result as its own item (for example SK=<run sk>#CASE#<idx>), or keep tighter truncation on the run item. The system prompt snapshot is worth keeping, but it does not have to sit on the run item. Storing it once per agent version (for example SK=AGENT#<id>#V<version>) lets every run of that version reference it. The agent's existing history only records version, time and author, not the prompt.

2. Keep the Observability page in English and keep the existing ledger view.
The rest of the portal is English, and this is a public sample. Please also keep the columns and hints that were removed from the invocation table (User, Turns, the per-source breakdown, the trace log group hint). They are what operators use today.

3. Load the three data sources independently.
Promise.all([listEvalRuns, getObservabilityStats, listInvocations]) means an eval API error now blanks the infrastructure metrics as well. Before this change they loaded separately.

4. Re-run button on a read-only page.
"Re-run" starts real model calls (agent plus judge) with one click. Please remove it from this page (the Evaluation page already starts runs), or at least add a confirmation that states the call count.

5. Scripts can overwrite existing agents with the same name.
Agents are published by name, and an admin can republish an agent owned by someone else. If a deployment already has an rnd-assistant or oncall-triage-bot, running the script with --republish replaces its system prompt, and channels and schedules bound to that agent id follow. Please add a configurable name prefix (for example --prefix, default obs-demo-) to both scripts.

6. Ledger join in the scripts.
/observability/invocations returns at most 200 rows. A full run is around 96 case calls plus about 60 judge calls, so on a deployment with other traffic part of the run falls outside the window. run_observability_demo.py then exits 1 even though the run was fine. This may also explain why the summary in the description shows 67 calls for 79 cases. Please treat a missing ledger row as "not in window" rather than a failure, or filter the ledger by ref on the server side.

Should fix

• Show sample size next to pass rates. Some datasets have 6 to 8 cases, so a single case moves the rate by 12 to 17 points, and the "pass rate dropped N points" notice will fire on noise. Showing n (or only flagging a drop above a threshold of cases) would avoid that.
• Average score for json_exact runs. Scores are only 0 or 10, so the average in the run history table is just the pass rate again. Hide it for structured scoring.
• json_exact matching rules. Matching is case-sensitive and the whole answer must be a JSON object (only code fences are stripped). That is a reasonable strict contract, but please state it in docs/observability.md so dataset authors know Billing fails against billing.
• Implicit default for classification. A dataset with scenario="classification" and no scoring silently becomes json_exact on category. Please document this or require scoring explicitly for new datasets.
• Judge independence. The judge runs on the same agent-sdk kernel as the agents under test. Worth noting in the docs next to the existing "screening signals" caveat, since it is one likely reason the judge missed the invented deadline.

Possible follow-up (not needed for this PR)

This PR covers evaluation before release: fixed cases against a pinned agent version. The open question is how to see quality on production traffic after an agent ships. AgentCore Evaluations has online evaluation for that. It samples live sessions from the spans already in CloudWatch, runs built-in or custom evaluators (LLM-as-a-judge with custom instructions, or code-based via Lambda), and publishes scores as metrics under Bedrock-AgentCore/Evaluations, where they can be alarmed on next to latency and cost.

A good next step would be a short investigation:

1. Can online evaluation attach to the sdk kernel runtime as deployed here (spans are already delivered)?
2. Can the rubrics in run_difficult_customer_evals.py become custom evaluators, and can json_exact become a code-based evaluator?
3. Regional availability, and whether a custom judge can use a global. cross-region inference profile.

A design question to think about: where should this data live?

Today the Observability page reads only DynamoDB: the invocation ledger (PK=INVOCATION) and now eval runs. Meanwhile CloudWatch already holds AgentCore service metrics (AWS/Bedrock-AgentCore), sdk kernel spans in aws/spans, and container logs, and the page does not use any of it. So the platform effectively has two observability stores that are never reconciled. The ledger join in your scripts, and the 200-row window problem, both come from that split.

We are considering CloudWatch as the single source for telemetry (logs, metrics, traces), with DynamoDB kept for state the application reads back to make decisions. No decision yet; I would like your view as part of the follow-up. Some questions to frame it:

1. Telemetry or state? For each record type (invocation ledger, eval runs and per-case evidence, quota counters under PK=USAGE, audit records), is it something people and alarms look at, or something the application reads back to decide? Quota counting, for example, needs an atomic increment before each call, which CloudWatch cannot do.
2. Read path. Logs Insights is asynchronous, billed by bytes scanned, and limited to 100 concurrent queries per account, so the portal cannot query it on every page load. Which parts of the page could use metrics (GetMetricData over EMF), which need on-demand Insights queries (with field indexes on run_id / trace_id), and which need a precomputed view?
3. Eval results specifically. If pass rates were emitted as EMF metrics and per-case evidence went to structured logs keyed by run_id, what would still need to live in DynamoDB? Would that also remove the 400 KB problem in item 1?
4. Freshness and accuracy. CloudWatch ingestion lags by up to about a minute, and EMF delivery is at least once. Which current flows depend on read-after-write (your scripts do), and does any of this data need to be exact, for example for per-team cost allocation?

A short write-up (a page or two, in the follow-up issue or PR) answering these would be enough. It does not need to come with code.

I ran backend/tests/test_eval_scenarios.py (7/7 pass) and tsc -b on the PR head locally. Both are clean.

[zzkamzn]

Thanks for the careful review. Everything under "Must fix" and "Should fix" is addressed in five new commits on top of the original two. I added commits rather than rewriting history, so each item can be checked on its own. The fifth commit goes a step further on item 6, described at the end.

Must fix

1. DynamoDB item size. Fixed in f9cbc25.

• Each case's evidence is now its own item, written once. The partition is PK=EVALRUN#<run_id> and the sort key is SK=CASE#<idx>. The results list is no longer rewritten after every case.
• A META item in that partition points at the run summary, so get_run reads by id and no longer scans the recent window. That removes the 404-while-polling case as well.
• The summary item keeps status and counts only, plus a new evaluated count.
• The system prompt is stored once per agent version under PK=AGENTPROMPT, SK=<agent_id>#V<version>, as you suggested. It is written with a condition, so every run of that version shares one copy.
• list_runs returns summaries; get_run returns full evidence and the prompt. The Evaluation page now fetches a run's details when it is expanded.
• Runs written before this change have their results on the summary item, and they are still read.
• The tests cover the worst case: 20 CJK cases with 8,000-character answers. Every run-side item stays under 40 KB, and the largest is the prompt snapshot at about 48 KB in a moto run against real DynamoDB semantics.

2. English, and the existing ledger view. Fixed in 24b7d8c.

• ObservabilityPage.tsx is back to the version on main with a five-line addition and one changed line. The added lines are the import, a refresh key, the component, and an "Invocation ledger" heading. The changed line is the Refresh button: it now also refreshes the new section. The User and Turns columns, the per-source breakdown and the trace log group hint are unchanged.
• Application outcomes are a separate ApplicationOutcomes component, in English, placed between the stat tiles and the ledger table.

3. Independent loading. Fixed in 24b7d8c. The component loads evaluation runs itself and has its own error banner. An evaluation API error no longer touches the infrastructure metrics.

4. Re-run button. Removed in 24b7d8c. Runs are started from the Evaluation page only.

5. Name collisions. Fixed in b7552df.

• Both scripts take --prefix, default obs-demo-, for agent and dataset names.
• They also stop with an error if an agent with one of their names exists but was not created by them. This check uses the description the scripts set. A prefix alone would still overwrite an agent that happened to be named obs-demo-rnd-assistant.

6. Ledger join. Fixed in b7552df.

• run_observability_demo.py now reports how many case calls it found in the window instead of exiting 1.
• run_difficult_customer_evals.py only joins runs whose case calls are all in the window. The infrastructure count and the application count therefore cover the same cases, and the script lists any runs it left out.
• 8fec457 then removes the join altogether, as described under "One more change" below.
• On the 67 versus 79 question: that one was my description, not the window. All 67 joined calls were found in the ledger. The table covered seven datasets: four from the extreme round, plus oncall, rnd and sales from the hard round. The label said "support", which made it read as if hard-round support, 12 more cases, was included. It was not. I have corrected the description, and the script's new summary prints which runs it joined.

Should fix

All five are done in 24b7d8c and 5fbaa1e.

• Every pass rate shows its sample size, for example 75% (6/8). A drop is flagged only when it amounts to at least two cases.
• json_exact runs hide the mean score.
• The json_exact contract is documented: the whole answer is one JSON object, only a code fence is stripped, the field must be a string, and comparison is trimmed and case-sensitive. The docs give Billing vs billing as the example.
• I kept the implicit classification default so existing datasets keep working, documented it, and recommend passing scoring explicitly. The demo script now does.
• Judge independence is noted next to the screening-signal caveat, using the invented deadline as the example.

One more change (8fec457)

Item 6 came from evaluation results and execution metrics living in two places. Each evaluated case now records the execution facts of its own calls, for the agent call and for the judge call:

• ok
• duration
• turns
• cost
• runtime session id

The run summary aggregates them when it ends. The page shows that "infrastructure view" next to the pass rate for exactly the same cases, for example "20/20 agent calls returned ok, 6 of those answers broke the expectation". The difficult-customer script prints it without touching the ledger. The session id on a case is also session.id on its kernel spans, which links a case to its trace when delivery is on.

The same commit fixes a problem the review did not mention. A dataset's run history was found by filtering the latest 50 runs of all datasets, so after enough other runs the previous run of a dataset dropped out and the regression comparison silently disappeared. Each run now writes a pointer under PK=EVALDS#<dataset_id>, and GET /api/v1/evals/datasets/{id}/runs reads one dataset's history directly.

Verification

• Merged current main into the branch, including Add an agentcore_gateway model backend: AgentCore Gateway in front of LiteLLM, per-session credentials #40 and Optional AgentCore Gateway interface endpoint for runtime → gateway traffic #45–README: describe the model-config save guard from #49 #51, with no conflicts. I merged rather than rebased so the commit hashes above stay valid.
• backend/tests/test_eval_scenarios.py: 13 tests pass, including 6 new storage tests. ruff is clean on the changed Python files.
• tsc -b and vite build pass.
• Redeployed to the same us-east-1 stack and re-ran both scripts with the new layout:
  • Demo benchmark, 2 runs and 21 cases: 21/21 agent calls returned ok, and 1 answer failed.
  • Hard round, 5 runs and 67 cases: 67/67 returned ok, and 8 answers broke a rule.
  • Extreme round, 4 runs and 29 cases: 29/29 returned ok, and 5 broke a rule.
  • Every case is in the comparison this time. Nothing is joined against the ledger, so nothing is "outside the window".

Found along the way, fixed separately

Answering your design question meant inventorying every DynamoDB record and every CloudWatch write. That turned up problems older than this PR, so they are in their own PRs and independent of #44:

• Find pipeline runs by id instead of scanning the latest 50 #52: pipeline runs could only be found among the latest 50, so run_sync reported completed runs as failed.
• Give expiring DynamoDB records a TTL #53: nothing in the table expired. Records now get a DynamoDB TTL. This includes one gateway grant row per headless call that was never removed.
• Keep large pipeline runs and service-entry results under the DynamoDB item limit #54: pipeline runs and service-entry results could exceed the 400 KB item limit.
• Cap runtime log retention and stop logging prompt text in the kernel #55: runtime log groups never expired, and the kernel logged prompt text into them.

Follow-ups

I opened #59 for online evaluation and the CloudWatch versus DynamoDB question. It has the investigation plan and a first pass at your four questions. One thing to flag there: the "spans are already delivered" premise depends on the deployment. agent_observability defaults to false and the base image emits no spans, so the stack I tested on has none yet. Enabling it is step one of the investigation.

[odinwang]

Landed via the internal sync in #63.