feat(eval): add read-only batch-evaluation CLI (get, list) - #1924
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## refactor #1924 +/- ##
============================================
+ Coverage 95.74% 95.78% +0.03%
============================================
Files 291 296 +5
Lines 16234 16431 +197
============================================
+ Hits 15544 15738 +194
- Misses 690 693 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| return { detail }; | ||
| } | ||
|
|
||
| try { |
There was a problem hiding this comment.
Don't we need to wait for this to become available?
There was a problem hiding this comment.
Get is intentionally not polling because a batch-evaluation job can run up to 4 hours. Blocking a CLI command for that long is rough, so get returns immediately and only pulls the CloudWatch results when the job is already terminal. I did go back and forth on a --wait but decided against it for now: customers can wrap get in a bash loop for the same effect without us owning the polling UX. Happy to add it if you want.
|
|
||
| // warnCloudWatchFailure writes a non-fatal advisory to stderr with a link to the | ||
| // CloudWatch results, so the machine-readable job status on stdout stays clean. | ||
| function warnCloudWatchFailure( |
There was a problem hiding this comment.
Might be nice to have a utility for writing warnings like this in the IO package. Or maybe a logger.
There was a problem hiding this comment.
Shifted it into io package. I would like to the customer to see this error without setting up a logger themselves
| @@ -0,0 +1,5 @@ | |||
| [ | |||
There was a problem hiding this comment.
batch-evaluation.test.tsx uses TestCoreClient, not fixtureFactories, so RECORD=1 cannot regenerate it. I’d either:
- Move real recordings to src/handlers/eval/batch-evaluation/fixtures and wire a fixture-backed command-flow test, or
- Keep it near the core parser test but name/document it explicitly as static test data, not a record/replay fixture.
Unless is there another reason this fixture is in the core?
There was a problem hiding this comment.
No this is mistake. I'll shift to /fixtures and wire a fixture backed tests.
There was a problem hiding this comment.
Just following up since this is still under src/core/__fixtures__ and the command-flow test still uses TestCoreClient. As it stands, RECORD=1 still cannot regenerate this recording and the new createLogsClient fixture seam is not exercised.
| const response = await logs.send( | ||
| new GetLogEventsCommand({ | ||
| logGroupName, | ||
| logStreamName: resultLogStreamName(batchEvaluationId), | ||
| startFromHead: true, | ||
| }), | ||
| ); |
There was a problem hiding this comment.
Can we paginate this call? GetLogEvents returns at most 1 MB/10,000 events and provides nextForwardToken. A single request silently returns incomplete results for larger jobs. I think we should probably consume pages until the forward token stops changing.
There was a problem hiding this comment.
Nice catch, I'll do that.
There was a problem hiding this comment.
Thanks, this handles normal pagination. I have one more small nit regarding pagination, but can be done in a followup. I was wondering if hitting MAX_RESULT_PAGES should flow back as resultsError instead of returning the accumulated array as complete. GetLogEvents is limited by 1 MB as well as 10,000 events, so large explanations can make 100 pages much less than roughly one million results. Right now the cap only writes to the file logger, while get --json succeeds with partial results and no customer-visible warning. Would it make sense to surface truncation through the same warning path as other CloudWatch failures?
| const response = await logs.send( | ||
| new GetLogEventsCommand({ | ||
| logGroupName, | ||
| logStreamName: resultLogStreamName(batchEvaluationId), |
There was a problem hiding this comment.
Should we pass cw.logStreamName into readEvaluationResults instead of deriving it from the batch evaluation ID? StartBatchEvaluationRequest does not accept an output destination, while GetBatchEvaluation returns the service-selected logGroupName and logStreamName. The observed run-${batchEvaluationId} format is not part of the SDK contract, so using the returned value avoids querying the wrong stream if that naming changes. Just a thought, definitely non-blocking.
There was a problem hiding this comment.
Nice catch, I'll do that.
c9d15ad to
40fa67c
Compare
Adds `agentcore eval batch-evaluation get` and `list` (read-only slice of the
imperative batch-evaluation commands).
- get --id [--disable-cw-results]: returns the service job and, by default, the
per-session results read from its CloudWatch output stream once terminal.
getBatchEvaluation returns { detail, resultsError? } — a CloudWatch read
failure is surfaced (io.warn on stderr), never hides the job status.
- list [--max-results/--next-token]: service-side pagination, page returned
verbatim (no client-side filtering).
Core:
- CloudWatch Logs client wired into AwsClients (logs() accessor + factory).
- core/batchEvaluationResults.tsx: reads + parses the result stream, following
GetLogEvents pagination to completion; uses the service-returned log
group/stream from outputConfig (no derived name). Parser keeps evaluation
level + session/trace/span scope (the old CLI flattened them); field names
verified against a recorded live stream.
- io.warn() helper for user-facing stderr advisories.
Tests: parser unit + real recorded-event replay, handler command-flow, io.warn.
Verified live against the exploratory account.
40fa67c to
fbb0cee
Compare
What
Adds the read-only slice of the imperative
batch-evaluationcommands.Commands
eval batch-evaluation get--id <id>(required)--disable-cw-results— skip CloudWatch retrieval, return service-side metadata onlyCOMPLETED/COMPLETED_WITH_ERRORS/FAILED/STOPPED) it reads per-session results from the CloudWatch output log stream. A CloudWatch failure warns on stderr and never hides or replaces the job status (stdout stays clean JSON).eval batch-evaluation list--max-results/--next-token— service-side pagination. Returns the service page verbatim (no client-side filtering).How
AwsClients—logs()accessor +createLogsClientfactory, injected at the app edge and in the test fixture seam. Region-only config, mirroring the IAM-client pattern (the AgentCore endpoint override must not leak onto CloudWatch).EvalClient(data plane) —getBatchEvaluation,listBatchEvaluations, andgetFullBatchEvaluationResults(batchEvaluationId, logGroupName, options)(derives the per-job result streamrun-<batchEvaluationId>). Batch evaluation lives on@aws-sdk/client-bedrock-agentcore, not the control plane.level(aws.bedrock_agentcore.evaluation_level) and scope ids (session.id,traceId), read verbatim from the recorded stream shape; a malformed log line is warned and skipped, never fatal.Example:
getdefault vs--disable-cw-resultsReal output from a freshly-run batch evaluation (
GTProbe2, COMPLETED, 9 sessions, 2 evaluators — one GT-requiring, one reference-free).Default — fetches per-session results from the CloudWatch output stream (showing first 4 of 18 results):
{ "batchEvaluationId": "GTProbe2_1786034545579-8ffefc851e", "batchEvaluationName": "GTProbe2_1786034545579", "status": "COMPLETED", "createdAt": "2026-08-06T16:42:26.468Z", "evaluators": [ { "evaluatorId": "Builtin.Correctness" }, { "evaluatorId": "Builtin.Helpfulness" } ], "dataSourceConfig": { "cloudWatchLogs": { "serviceNames": ["repro1427_ReproAgent.DEFAULT"], "logGroupNames": ["/aws/bedrock-agentcore/runtimes/repro1427_ReproAgent-zqNEUl8I0f-DEFAULT"], "filterConfig": { "timeRange": { "startTime": "2026-07-29T00:00:00.000Z", "endTime": "2026-07-30T00:00:00.000Z" } } } }, "outputConfig": { "cloudWatchConfig": { "logGroupName": "/aws/bedrock-agentcore/evaluations/batch-evaluations/results/default", "logStreamName": "run-GTProbe2_1786034545579-8ffefc851e" } }, "evaluationResults": { "numberOfSessionsCompleted": 9, "numberOfSessionsInProgress": 0, "numberOfSessionsFailed": 0, "totalNumberOfSessions": 9, "numberOfSessionsIgnored": 0, "evaluatorSummaries": [ { "evaluatorId": "Builtin.Helpfulness", "statistics": { "averageScore": 0.83 }, "totalEvaluated": 9, "totalFailed": 0 }, { "evaluatorId": "Builtin.Correctness", "statistics": { "averageScore": 1 }, "totalEvaluated": 9, "totalFailed": 0 } ] }, "updatedAt": "2026-08-06T16:43:30.696Z", "results": [ { "evaluatorId": "Builtin.Correctness", "level": "Trace", "sessionId": "baseline-session-3-1785350958-aaa...", "traceId": "6a6a4b30737b856660b6a2c37e1c4b93", "score": 1, "label": "Correct", "explanation": "The user asked to add 17 and 25 using the add_numbers tool..." }, { "evaluatorId": "Builtin.Helpfulness", "level": "Trace", "sessionId": "baseline-session-3-1785350958-aaa...", "traceId": "6a6a4b30737b856660b6a2c37e1c4b93", "score": 0.83, "label": "Very Helpful", "explanation": "The user's goal was simple and clear..." }, { "evaluatorId": "Builtin.Correctness", "level": "Trace", "sessionId": "baseline-session-1-1785350935-aaa...", "traceId": "6a6a4b1806f680cd2ed2308c1062be9b", "score": 1, "label": "Correct", "explanation": "The user asked to use the add_numbers tool..." }, { "evaluatorId": "Builtin.Helpfulness", "level": "Trace", "sessionId": "baseline-session-1-1785350935-aaa...", "traceId": "6a6a4b1806f680cd2ed2308c1062be9b", "score": 0.83, "label": "Very Helpful", "explanation": "The user's goal was simple and clear..." } ] }--disable-cw-results— skips the CloudWatch call; identical service-side metadata, noresultskey:{ "batchEvaluationId": "GTProbe2_1786034545579-8ffefc851e", "batchEvaluationName": "GTProbe2_1786034545579", "status": "COMPLETED", "createdAt": "2026-08-06T16:42:26.468Z", "evaluators": [ { "evaluatorId": "Builtin.Correctness" }, { "evaluatorId": "Builtin.Helpfulness" } ], "dataSourceConfig": { "cloudWatchLogs": { "serviceNames": ["repro1427_ReproAgent.DEFAULT"], "logGroupNames": ["/aws/bedrock-agentcore/runtimes/repro1427_ReproAgent-zqNEUl8I0f-DEFAULT"], "filterConfig": { "timeRange": { "startTime": "2026-07-29T00:00:00.000Z", "endTime": "2026-07-30T00:00:00.000Z" } } } }, "outputConfig": { "cloudWatchConfig": { "logGroupName": "/aws/bedrock-agentcore/evaluations/batch-evaluations/results/default", "logStreamName": "run-GTProbe2_1786034545579-8ffefc851e" } }, "evaluationResults": { "numberOfSessionsCompleted": 9, "numberOfSessionsInProgress": 0, "numberOfSessionsFailed": 0, "totalNumberOfSessions": 9, "numberOfSessionsIgnored": 0, "evaluatorSummaries": [ { "evaluatorId": "Builtin.Helpfulness", "statistics": { "averageScore": 0.83 }, "totalEvaluated": 9, "totalFailed": 0 }, { "evaluatorId": "Builtin.Correctness", "statistics": { "averageScore": 1 }, "totalEvaluated": 9, "totalFailed": 0 } ] }, "updatedAt": "2026-08-06T16:43:30.696Z" }Testing
src/core/eval.batch.test.ts— parser unit tests over a multi-level stream (Session + Trace + non-JSON control line) and a replay of real recorded events (src/core/__fixtures__/batch-eval-result-log-events.json), so a future service-side shape change surfaces here.src/handlers/eval/batch-evaluation/batch-evaluation.test.tsx— handler tests through the real router: CloudWatch-on-terminal default,--disable-cw-results, non-terminal skip, CloudWatch-failure-warns-but-status-intact, and list pagination.RECORD=1run against the live API.