update

EVALUATIONS.md

@@ -197,6 +197,82 @@ client.evaluations.datasets.builder(name="Support Agent v2").add_case(
 
 ---
 
+### Agent trajectory and retrieval-context checks (deterministic, no judge spend)
+
+Two per-case expectations are scored server-side against the case's **linked trace** (return
+`{"output": ..., "trace_id": span.trace_id}` from your agent function, with the call wrapped in
+`client.tracer.trace(..., sync=True)`):
+
+```python
+client.evaluations.datasets.builder(name="Agent checks").add_case(
+    query="Refund order 4412 and email a confirmation.",
+    expected_results="Refund issued and confirmation sent.",
+    # The tool calls a correct run should make, matched against the trace's REAL calls.
+    # Modes (agentevals semantics): strict | unordered | superset | subset.
+    expected_tools=["lookup_order", "issue_refund", "send_email"],
+    trajectory_match_mode="strict",
+    # What a correct retriever should have fetched - compared to the actual retrieved context
+    # with token Jaccard. Catches retriever regressions even when the answer text is identical.
+    expected_retrieval_context=["Refunds: 30 days, no restocking fee."],
+)
+```
+
+Each produces a scorer row on the result (`Trajectory match (<mode>)` pass/fail, and
+`Context match (jaccard)` 0-1). Both are deterministic: no LLM judge call, no spend, and they
+run on every result that carries the needed evidence.
+
+---
+
+### Dataset splits (cheap PR runs vs. nightly full runs)
+
+Tag cases with named subsets and run just one subset:
+
+```python
+builder.add_case(query="smoke case", splits=["smoke"])
+builder.add_case(query="full-only case")
+
+client.evaluations.run(dataset_id, subject, split="smoke").execute(my_fn).finalize()
+```
+
+Original case indexes are preserved, so per-case comparisons line up between a split run and a
+full run. The connector-driven dashboard run accepts the same `split`.
+
+---
+
+### Concurrency and output reuse
+
+```python
+run.execute(my_fn, concurrency=4)                    # thread-pooled agent calls, ordered submission
+run.execute(my_fn, reuse_outputs_from="run_abc123")  # replay a previous run's outputs
+```
+
+`reuse_outputs_from` replays the recorded output for every case whose query text is unchanged
+(errored rows and changed/new cases run normally) and the judge re-scores everything with THIS
+run's grading config - which makes iterating on scorers essentially free. Reused results carry
+`metadata.reusedFromRun`.
+
+Interrupted runs resume: `execute()` asks the engine which idempotency keys were already
+accepted and skips those cases, so a crash or a failed batch (which now raises
+`EvaluationSubmissionError` instead of finishing silently empty) never re-pays for finished
+work - just call `execute()` again on the same context.
+
+---
+
+### Human review queue (label-and-calibrate from code)
+
+```python
+item = client.monitor.review_queue.queue(trace_id, note="spot-check this")
+for item in client.monitor.review_queue.list(status="pending"):
+    client.monitor.review_queue.label(item.id, "bad", corrected_score=2, note="hallucinated")
+```
+
+Labels (with the judge's own score for the same trace) feed per-scorer calibration
+(`client.monitor.judge_scorers.calibration(scorer_id)`) and become judge-tuning evidence.
+Project-level numbers come from `client.monitor.calibration(window="7d")`, whose exact wire
+keys are `comparedCount`, `agreementRate`, `falsePositiveRate`, `falseNegativeRate`.
+
+---
+
 ### LLM Judge Scorers - reusable grading configs
 
 By default, a dataset runs against the grading config it was created with (`number_of_requests`, `acceptance_criteria`, similarity metrics, etc. - see above). If you want to grade the **same dataset** against **different configs** (e.g. a strict config vs. a lenient one, or reuse one config across many datasets), create a standalone **LLM Judge Scorer** and pass its id to `.run()`.
@@ -285,13 +361,13 @@ Calibration, tuning and the live-scoring history hang off the same scorer id - t
 ```python
 client.monitor.judge_scorers.calibration(scorer.id, window="7d")   # verdicts vs. recorded ground truth
 proposal = client.monitor.judge_scorers.tune(scorer.id)            # LLM call, slow
-client.monitor.judge_scorers.validate_tuning(scorer.id, proposal)  # re-judge with candidate criteria
-client.monitor.judge_scorers.publish_tuning(scorer.id, proposal)   # write it onto the rubric
+verdict = client.monitor.judge_scorers.validate_tuning(scorer.id, proposal)  # re-judge with candidate criteria
+client.monitor.judge_scorers.publish_tuning(scorer.id, proposal, validation=verdict)  # write it onto the rubric
 client.monitor.judge_scorers.ratings(scorer.id, window="7d")       # -> list[OnlineEvaluatorRatingPoint]
 client.monitor.judge_scorers.events(scorer.id, window="7d")        # -> list[OnlineEvaluatorEvent]
 ```
 
-Those six cover live-traffic scoring, so calling them on an offline-only scorer raises `AgentXJudgeScorersError` naming the fix (`update(scorer_id, online={"enabled": True})`). `publish_tuning` writes to the shared rubric, so it applies everywhere the scorer is used: online scoring, offline dataset runs and the playground alike.
+Those six cover live-traffic scoring, so calling them on an offline-only scorer raises `AgentXJudgeScorersError` naming the fix (`update(scorer_id, online={"enabled": True})`). `publish_tuning` writes to the shared rubric, so it applies everywhere the scorer is used: online scoring, offline dataset runs and the playground alike. Publish is provenance-gated: the engine refuses an unvalidated publish, and a measured regression, unless you pass `force=True`; the validation verdict is stamped into the rubric's version history.
 
 #### Engine compatibility (self-host)
 
@@ -854,12 +930,9 @@ Your callable can return any of:
 | `dict` with `"output"` key | Output text from `output`, rest stored as metadata |
 | `EvaluationResult` | Full control - pass rating, justification, trace, timings |
 
-### Security and redaction
-
-The SDK automatically scrubs secrets from outputs and metadata before uploading:
-- `sk-...` API keys
-- Bearer tokens
-- Authorization headers
-- Password-like fields
+### What gets uploaded
 
-Raw agent outputs, prompts, and CoT reasoning are **never uploaded** - only the text response, metadata you explicitly include, and optional observable trace summaries.
+The SDK uploads exactly what your callable returns: the output text, any metadata you include,
+and (when tracing is enabled) the trace you instrumented. **The SDK does not scrub or redact
+anything** - if your agent's output or metadata can contain secrets, redact them in your own
+code before returning, or keep them out of the returned payload entirely.

agentx/evaluations/client.py

@@ -60,6 +60,13 @@ class AgentXValidationError(AgentXEvaluationsError):
     pass
 
 
+class EvaluationSubmissionError(AgentXEvaluationsError):
+    """A result batch could not be submitted (after one retry). The run is left unfinalized;
+    re-running execute() on the same context resumes past already-submitted cases."""
+
+    pass
+
+
 def _resolve_scorer_id(scorer_id: Optional[str], evaluation_settings_id: Optional[str]) -> Optional[str]:
     """One grader, two spellings: ``scorer_id`` is the post-consolidation name for what the wire
     still calls ``evaluationSettingsId`` (the ids are identical by design). Both kwargs are
@@ -297,10 +304,11 @@ def init_run(
         python_version: Optional[str] = None,
         scorer_id: Optional[str] = None,
         evaluation_settings_id: Optional[str] = None,
+        split: Optional[str] = None,
     ) -> EvaluationRun:
         """``scorer_id`` names the LLM Judge Scorer grading this run (its id doubles as the
         wire's ``evaluationSettingsId``). ``evaluation_settings_id`` is the pre-consolidation
-        alias and keeps working."""
+        alias and keeps working. ``split`` records the named case subset this run covers."""
         from agentx.version import VERSION
 
         grader_id = _resolve_scorer_id(scorer_id, evaluation_settings_id)
@@ -318,6 +326,8 @@ def init_run(
         }
         if grader_id:
             payload["evaluationSettingsId"] = grader_id
+        if split:
+            payload["split"] = split
         data = self._request("POST", "/runs", json=self._with_workspace(payload))
         return EvaluationRun(**data)
 
@@ -436,6 +446,13 @@ def get_missing_results(self, run_id: str) -> List[Dict[str, Any]]:
         data = self._request("GET", f"/runs/{run_id}/missing-results")
         return data if isinstance(data, list) else data.get("missing", [])
 
+    def get_submitted_keys(self, run_id: str) -> List[str]:
+        """Idempotency keys this run has already accepted - what execute() uses to resume a
+        crashed or interrupted run without re-running (and re-paying for) finished cases."""
+        data = self._request("GET", f"/runs/{run_id}/missing-results")
+        keys = data.get("submittedKeys", []) if isinstance(data, dict) else []
+        return [k for k in keys if isinstance(k, str)]
+
     # ------------------------------------------------------------------
     # Self-host analysis fallback
     #

agentx/evaluations/datasets.py

@@ -105,9 +105,15 @@ def add_case(
         expected_tools: Optional[List[str]] = None,
         trajectory_match_mode: str = "strict",
         expected_retrieval_context: Optional[Union[str, List[str]]] = None,
+        splits: Optional[List[str]] = None,
     ) -> "DatasetBuilder":
         """Add a case. `judge_guideline` is optional grading guidance specific to this question.
 
+        `splits` tags this case with named subsets (e.g. ``["smoke"]``): a run started with
+        ``client.evaluations.run(dataset_id, subject, split="smoke")`` executes only the tagged
+        cases (original case indexes are preserved, so per-case comparisons still line up with
+        full runs). An untagged case belongs to no split and only runs in full runs.
+
         `expected_tools` declares the tool calls a correct run of this case should make. When a
         result links its trace (return `{"output": ..., "trace_id": span.trace_id}` from the
         agent function), the engine matches the trace's actual tool-call sequence against it and
@@ -149,6 +155,8 @@ def add_case(
             main["expectedTrajectory"] = {"tools": expected_tools, "mode": trajectory_match_mode}
         if expected_retrieval_context:
             main["expectedRetrievalContext"] = expected_retrieval_context
+        if splits:
+            main["splits"] = splits
         self._payload["questions"].append(
             {
                 "main_question": main,

agentx/evaluations/models.py

@@ -52,6 +52,9 @@ class TestCase(BaseModel):
     expected_delegations: Optional[List[str]] = Field(default=None, alias="expectedDelegations")
     judge_guideline: Optional[str] = Field(default=None, alias="judgeGuideline")
     smoke_test: Optional[SmokeTestSettings] = Field(default=None, alias="smokeTest")
+    # Named subsets this case belongs to (e.g. ["smoke"], ["full", "regression"]).
+    # ``run(dataset_id, split="smoke")`` runs only cases tagged with that split.
+    splits: Optional[List[str]] = None
 
     class Config:
         populate_by_name = True