diff --git a/fern/apis/api/openapi-overrides.yml b/fern/apis/api/openapi-overrides.yml index 41a047508..47d4adcfc 100644 --- a/fern/apis/api/openapi-overrides.yml +++ b/fern/apis/api/openapi-overrides.yml @@ -998,6 +998,16 @@ components: A record of an eval execution, including its target, status, results, costs, completion details, and lifecycle timestamps. properties: + results: + description: >- + Results for this individual Eval. Check them after status is ended. + An Eval that finishes normally contains one result; it passes only + when all judged checkpoints pass. Grouping multiple Evals requires your + own orchestration, not an Eval suite. + cost: + description: The cost of this Eval run in USD. + costs: + description: The cost breakdown for this Eval run. id: description: The unique identifier for the eval run. orgId: diff --git a/fern/assistants/examples/appointment-scheduling.mdx b/fern/assistants/examples/appointment-scheduling.mdx index c4a433462..53c4360d3 100644 --- a/fern/assistants/examples/appointment-scheduling.mdx +++ b/fern/assistants/examples/appointment-scheduling.mdx @@ -258,6 +258,16 @@ Use the Google Calendar integration for availability and booking, or your own AP ## 5. Test and validate +Use sandbox calendar records and test identities. Don't create, change, or cancel real customer appointments during testing. + +| Check | Coverage | +| --- | --- | +| Evals | Correct booking arguments when all required details are known; ask when timezone is missing; don't confirm after a tool error. | +| Simulations | A complete booking, an unavailable slot with an alternative, and rescheduling without losing the original appointment if the change fails. | +| Controlled calls | Listen for date and time clarity, and verify the intended changes actually exist in the sandbox calendar. | + +Repeat critical scenarios before release. A tool mock checks behavior under a supplied result, not the calendar integration. See [plan test coverage](/test/plan-test-coverage). + Create a phone number and assign your assistant. See [Phone calls quickstart](/quickstart/phone). @@ -274,4 +284,3 @@ Use the Google Calendar integration for availability and booking, or your own AP - **Tools**: [Google Calendar](/tools/google-calendar), [Custom Tools](/tools/custom-tools) - **Structured outputs**: [Extract structured data](/assistants/structured-outputs-quickstart) - **Multichannel**: [Web integration](/quickstart/web) - diff --git a/fern/assistants/examples/support-escalation.mdx b/fern/assistants/examples/support-escalation.mdx index 0dd42f5af..f3e79460a 100644 --- a/fern/assistants/examples/support-escalation.mdx +++ b/fern/assistants/examples/support-escalation.mdx @@ -705,6 +705,16 @@ Always be professional and efficient in your support.""" ## 4. Test Your Support Escalation System +Use dedicated test accounts and transfer destinations staffed by your team. The scripts below create calls for manual testing; they don't simulate callers or assert that escalation worked. + +| Check | Coverage | +| --- | --- | +| Evals | Escalate when policy requires it, avoid unnecessary escalation, and request the correct destination with the required context. | +| Simulations | Follow an unresolved issue through escalation, including a failed or unavailable destination and a safe next step. | +| Controlled calls | Verify the real transfer connects, the recipient receives the needed context, and the audio remains usable. | + +Repeat critical checks and turn routing failures into regression tests. See [test decisions with Evals](/test/evals-best-practices) and [test outcomes with Simulations](/test/simulations-best-practices). + diff --git a/fern/assistants/structured-outputs-examples.mdx b/fern/assistants/structured-outputs-examples.mdx index 36472a2f5..cb0ebb00a 100644 --- a/fern/assistants/structured-outputs-examples.mdx +++ b/fern/assistants/structured-outputs-examples.mdx @@ -1215,6 +1215,8 @@ Always test your structured outputs with these scenarios: 4. **Edge cases** - Boundary values, special characters 5. **Real conversations** - Actual call recordings or transcripts +Valid JSON doesn't guarantee a correct extraction or judgment. Compare each output with expected values from human-reviewed examples, including known successes and failures. For outcome judgments, record what evidence is available: a reported booking isn't proof of a calendar update. Use synthetic values or remove personal information from reused calls. Follow [judge calibration and result review](/test/run-and-maintain-tests#review-more-than-the-pass-or-fail-label). + ### Monitoring checklist Track these metrics for production deployments: @@ -1291,4 +1293,4 @@ When retrieving call data via API: - [Structured outputs overview](/assistants/structured-outputs) - Main documentation - [Quickstart guide](/assistants/structured-outputs-quickstart) - Get started quickly - [API reference](/api-reference#structured-output) - Complete API documentation -- [JSON Schema specification](https://json-schema.org/) - JSON Schema standard \ No newline at end of file +- [JSON Schema specification](https://json-schema.org/) - JSON Schema standard diff --git a/fern/assistants/structured-outputs-quickstart.mdx b/fern/assistants/structured-outputs-quickstart.mdx index 9cd34874d..90b4dc7a4 100644 --- a/fern/assistants/structured-outputs-quickstart.mdx +++ b/fern/assistants/structured-outputs-quickstart.mdx @@ -912,6 +912,7 @@ Common validation patterns for reliable extraction: - Use enums for categorical data to ensure consistency - Add descriptions to help the AI understand context - Test with real conversations before production use +- Compare extracted values and judgments with human-reviewed examples. A valid schema checks the output's shape, not whether its values are true. - Monitor extraction success rates and iterate on schemas diff --git a/fern/customization/multilingual.mdx b/fern/customization/multilingual.mdx index fe4dc192f..4c372aaaa 100644 --- a/fern/customization/multilingual.mdx +++ b/fern/customization/multilingual.mdx @@ -406,6 +406,14 @@ Configure greeting messages that work across multiple languages. Validate your configuration with different languages and scenarios. +| Check | Coverage | +| --- | --- | +| Evals | Check the next response or routing decision in each supported language, including when the caller changes language. These checks don't test speech recognition. | +| Voice Simulations | Run representative customer journeys in each language and with mixed-language input. Repeat critical cases and listen to recordings. | +| Controlled calls | Ask native speakers to review understanding, pronunciation, and turn-taking on the real phone path. Synthetic voices aren't complete accent or noise coverage. | + +Use [plan test coverage](/test/plan-test-coverage) to prioritize the languages and journeys your customers depend on. Review both failed and passing calls. + 1. Use the **Test Assistant** feature in your dashboard diff --git a/fern/observability/evals-advanced.mdx b/fern/observability/evals-advanced.mdx index 23c8b1351..a3ceb2eba 100644 --- a/fern/observability/evals-advanced.mdx +++ b/fern/observability/evals-advanced.mdx @@ -74,6 +74,8 @@ Ensure fixes and updates don't break existing functionality. **Purpose:** Validate that known issues stay fixed and features keep working. +Supply the context that makes the expected action valid. The minimal date example below assumes a date-only booking tool with no other prerequisites. For relative dates, fix the reference date and timezone in the mock context. Add a paired case where missing or ambiguous information requires clarification instead of a tool call. + 1. Create evaluation named with "Regression: " prefix @@ -83,7 +85,7 @@ Ensure fixes and updates don't break existing functionality. Example: - Name: "Regression: Date Parsing Bug #1234" - - Description: "Verify dates like '3/15' parse correctly after bug fix" + - Description: "Verify an explicitly dated request uses the correct booking date" @@ -93,12 +95,12 @@ curl -X POST "https://api.vapi.ai/eval" \ -H "Content-Type: application/json" \ -d '{ "name": "Regression: Date Parsing Bug #1234", - "description": "Verify dates like 3/15 are parsed correctly after fix", + "description": "Verify an explicitly dated request uses the correct booking date", "type": "chat.mockConversation", "messages": [ { "role": "user", - "content": "Book me for 3/15" + "content": "Book me for March 15, 2027" }, { "role": "assistant", @@ -107,7 +109,7 @@ curl -X POST "https://api.vapi.ai/eval" \ "toolCalls": [{ "name": "bookAppointment", "arguments": { - "date": "2025-03-15" + "date": "2027-03-15" } }] } @@ -259,8 +261,10 @@ Test boundary conditions and unusual inputs. - **Input boundaries:** Empty, maximum length, special characters - **Data formats:** Invalid dates, malformed phone numbers, unusual names -- **Conversation patterns:** Interruptions, topic changes, contradictions -- **Timing:** Very fast responses, long pauses, timeout scenarios +- **Conversation decisions:** Topic changes, contradictions, requests to stop +- **Mock tool results:** Error or timeout messages and the next safe response + +Evals don't exercise audio or elapsed-time behavior. A mocked timeout message tests the assistant's response to that message, not a real timeout. Use Voice Simulations and recordings to investigate speech and turn-taking, then controlled calls to verify the real phone path. Synthetic callers don't reproduce every interruption, silence, or background-noise condition reliably. ## Testing patterns @@ -305,7 +309,7 @@ Validate ideal user journeys where everything works correctly. }, { "role": "user", - "content": "Next Monday at 2pm please" + "content": "January 20, 2027 at 2pm Pacific time please" }, { "role": "assistant", @@ -315,7 +319,7 @@ Validate ideal user journeys where everything works correctly. { "name": "bookAppointment", "arguments": { - "date": "2025-01-20", + "date": "2027-01-20", "time": "14:00" } } @@ -329,8 +333,15 @@ Validate ideal user journeys where everything works correctly. { "role": "assistant", "judgePlan": { - "type": "regex", - "content": ".*(confirmed|booked).*APT-12345.*" + "type": "ai", + "model": { + "provider": "openai", + "model": "gpt-4.1", + "messages": [{ + "role": "system", + "content": "Context: {{messages}}. Pass only if the last assistant message accurately reports the successful tool result and shares confirmation ID APT-12345. Otherwise fail. Respond only with pass or fail." + }] + } } } ] @@ -419,7 +430,7 @@ Test how your assistant handles failures gracefully. } ``` -**API timeout simulation:** +**Response to a mocked timeout message:** ```json { @@ -583,7 +594,7 @@ Test behavior at or near rate limits: - Keep evaluations focused (5-10 turns max). + Include only the context and checkpoints needed to test the decision. Split complex scenarios into multiple targeted tests. @@ -599,14 +610,14 @@ Choose the right judge type for each scenario: - Critical business data (confirmation IDs, totals, dates) - Tool call validation with specific arguments - Compliance-required exact wording - - Success/failure status messages + - Fixed response text required by your policy - **Example:** Booking confirmation ID must be exact + **Example:** A policy requires this exact disclosure ```json { "judgePlan": { "type": "exact", - "content": "Your confirmation ID is APT-12345" + "content": "This call may be recorded." } } ``` @@ -635,10 +646,9 @@ Choose the right judge type for each scenario: - Semantic meaning validation - Tone and sentiment evaluation - Contextual appropriateness - - Complex multi-factor criteria - - Helpfulness assessment + - One clearly defined decision with several valid phrasings - **Example:** Validate polite rejection + **Example:** Check rejection of an unsupported request ```json { "judgePlan": { @@ -648,7 +658,7 @@ Choose the right judge type for each scenario: "model": "gpt-4o", "messages": [{ "role": "system", - "content": "PASS if response politely declines without being rude and offers alternative. Output: pass or fail" + "content": "Context: {{messages}}. Pass if the last assistant message declines the unsupported request. Fail if it agrees to carry it out. Output only pass or fail." }] } } @@ -691,7 +701,7 @@ other Evals run; your own automation must handle that decision. either manually or in your own automation. 3. **Keep conversations focused:** - Aim for 5-10 turns maximum. Split longer scenarios into multiple tests. + Keep only the context needed for the decision. Use separate Evals for independent decisions and Simulations for complete conversation outcomes. 4. **Batch related tests:** If you need to run a group in sequence, keep the Eval IDs in your own script @@ -772,111 +782,76 @@ Don't delete tests immediately when features change: ### CI/CD integration -Automate evaluation runs in your deployment pipeline. +This workflow runs saved Evals against an existing staging assistant. Add it to your own repository, not to your production deployment until you have verified its results. + +Set the repository secret `VAPI_API_KEY` and repository variables `STAGING_ASSISTANT_ID` and `REQUIRED_EVAL_IDS` (space-separated Eval IDs). Deploy the intended assistant configuration to staging before running it. This example checks saved staging state; it does not deploy a pull request's prompt changes. -**Basic workflow:** +To make this a release gate, run the job after your staging update and require it to pass before promoting the same configuration to production. ```yaml # .github/workflows/test-assistant.yml -name: Test Assistant Changes +name: Check required Evals on: - pull_request: - paths: - - "assistants/**" - - "prompts/**" + workflow_dispatch: jobs: run-evals: runs-on: ubuntu-latest + timeout-minutes: 30 + env: + VAPI_API_KEY: ${{ secrets.VAPI_API_KEY }} + STAGING_ASSISTANT_ID: ${{ vars.STAGING_ASSISTANT_ID }} + REQUIRED_EVAL_IDS: ${{ vars.REQUIRED_EVAL_IDS }} steps: - - name: Run critical evals + - name: Run and check required Evals + shell: bash run: | - # Run smoke tests - curl -X POST "https://api.vapi.ai/eval/run" \ - -H "Authorization: Bearer ${{ secrets.VAPI_API_KEY }}" \ - -d '{"evalId": "$SMOKE_TEST_ID", "target": {...}}' - - # Check results - # Fail build if tests fail + set -euo pipefail + : "${VAPI_API_KEY:?Set VAPI_API_KEY}" + : "${STAGING_ASSISTANT_ID:?Set STAGING_ASSISTANT_ID}" + : "${REQUIRED_EVAL_IDS:?Set REQUIRED_EVAL_IDS}" + read -r -a eval_ids <<< "$REQUIRED_EVAL_IDS" + (("${#eval_ids[@]}" > 0)) + + for eval_id in "${eval_ids[@]}"; do + payload=$(jq -n --arg id "$eval_id" --arg target "$STAGING_ASSISTANT_ID" \ + '{type:"eval", evalId:$id, target:{type:"assistant", assistantId:$target}}') + run=$(curl --fail-with-body -sS --connect-timeout 10 --max-time 30 \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d "$payload" https://api.vapi.ai/eval/run) + run_id=$(jq -er '.id | select(type == "string" and length > 0)' <<< "$run") + deadline=$((SECONDS + 300)) + + while [[ $(jq -r '.status' <<< "$run") != ended ]]; do + if ((SECONDS >= deadline)); then + echo "Timed out waiting for Eval $eval_id (run $run_id)" + exit 1 + fi + sleep 2 + run=$(curl --fail-with-body -sS --connect-timeout 10 --max-time 30 \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + "https://api.vapi.ai/eval/run/$run_id") + done + + if ! jq -e ' + .endedReason == "mockConversation.done" and + (.results | type == "array" and length > 0 and + all(.[]; .status == "pass")) + ' <<< "$run" > /dev/null; then + echo "Required Eval $eval_id failed or ended abnormally (run $run_id)" + exit 1 + fi + echo "Required Eval $eval_id passed (run $run_id)" + done ``` -**Advanced patterns:** +The example waits up to five minutes per Eval, plus any in-flight HTTP request. Adjust the deadlines for your workload. A timeout is a failed check, not a pass; inspect the run before retrying. The workflow doesn't cancel a remote run when it stops waiting. - - - Run full eval suite against staging before production deploy: - - ```bash - # Run all evals against staging assistant - for eval_id in $EVAL_IDS; do - run_result=$(curl -X POST "https://api.vapi.ai/eval/run" \ - -H "Authorization: Bearer $VAPI_API_KEY" \ - -d "{\"evalId\": \"$eval_id\", \"target\": {\"type\": \"assistant\", \"assistantId\": \"$STAGING_ASSISTANT_ID\"}}") - - # Check if passed - status=$(echo $run_result | jq -r '.results[0].status') - if [ "$status" != "pass" ]; then - echo "Eval $eval_id failed!" - exit 1 - fi - done - ``` - +Keep required regression Evals separate from improvement tests. Block release on any unexplained failure in a required check, including execution errors. Report improvement results separately so they don't dilute critical failures in an overall pass rate. - - Run multiple evals concurrently to speed up CI: - - ```bash - # Run evals in parallel - for eval_id in $EVAL_IDS; do - (curl -X POST "https://api.vapi.ai/eval/run" \ - -H "Authorization: Bearer $VAPI_API_KEY" \ - -d "{\"evalId\": \"$eval_id\", ...}" > results_$eval_id.json) & - done - wait - - # Aggregate results - for result_file in results_*.json; do - # Check each result - done - ``` - - -{" "} - - - Block deployment if test pass rate falls below threshold: ```bash # Calculate - pass rate total_tests=10 passed_tests=$(grep -c '"status":"pass"' - all_results.json) pass_rate=$((passed_tests * 100 / total_tests)) if [ - $pass_rate -lt 95 ]; then echo "Pass rate $pass_rate% below threshold 95%" - exit 1 fi ``` - - - - Run full regression suite nightly: - - ```yaml - # .github/workflows/nightly-regression.yml - on: - schedule: - - cron: '0 2 * * *' # 2 AM daily - - jobs: - regression-suite: - runs-on: ubuntu-latest - steps: - - name: Run regression tests - run: ./scripts/run-regression-suite.sh - - - name: Notify on failures - if: failure() - run: | - # Send Slack notification - # Create GitHub issue - ``` - - +One passing run isn't proof of reliability. Repeat critical or surprising Evals and investigate variation. For scheduled or parallel execution, retain the same completion and result checks for every run. Grouping, scheduling, and repetitions are your automation's responsibility, not a native Eval-suite feature. Add repeated Simulations and controlled calls to your [release checks](/test/run-and-maintain-tests). ## Advanced troubleshooting @@ -949,19 +924,13 @@ jobs: {" "} - **Problem:** Arguments have different types or extra fields **Solutions:** - - Check argument types: `"14:00"` (string) vs `14` (number) - Use partial - matching: omit `arguments` to match only function name - Normalize data - formats in tool implementation + Check whether the expected arguments match the tool's contract and the facts in the conversation. For example, `"14:00"` and `14` aren't interchangeable if the tool expects a time string. Fix the assistant when its arguments are wrong; fix the test when its expectation is wrong. Omit `arguments` only when the tool name alone is the requirement. Don't drop checks for dates, account IDs, or other important values just to get a pass. {" "} - **Problem:** Same response sometimes passes, sometimes fails **Solutions:** - - Make criteria more specific and binary - Add explicit examples of pass/fail - cases in prompt - Use temperature=0 for deterministic evaluation - Switch to - regex if pattern-based validation works + Give the judge one explicit pass/fail question and calibrate it against responses that reviewers agree should pass or fail. Lower temperature may reduce variation, but doesn't guarantee identical results. Rerun critical or surprising checks and review disagreements. Use exact matching or regex only when a fixed value or format proves the requirement; use separate checks for independent requirements such as correctness and tone. {" "} @@ -1014,18 +983,7 @@ When tests fail inconsistently: **Progressive validation:** -Build up complexity gradually: - -```json -// Step 1: Verify basic response -{"judgePlan": {"type": "regex", "content": ".+"}} - -// Step 2: Verify contains keyword -{"judgePlan": {"type": "regex", "content": ".*appointment.*"}} - -// Step 3: Verify exact format -{"judgePlan": {"type": "exact", "content": "Appointment confirmed"}} -``` +Use a non-empty-response check only to diagnose basic setup. Then restore the check that proves the requirement: the correct tool arguments, the response to a tool error, or another intended decision. Keywords or the phrase "Appointment confirmed" don't prove a booking succeeded. Don't use weakened diagnostic checks as release gates. ## Troubleshooting reference @@ -1044,7 +1002,7 @@ Build up complexity gradually: **When an eval fails, check:** - [ ] `endedReason` is "mockConversation.done" - [ ] Assistant works correctly in manual testing - [ ] Tool endpoints are - accessible - [ ] Validation criteria match actual behavior - [ ] Regex + accessible - [ ] Validation criteria match intended requirements - [ ] Regex patterns are properly escaped - [ ] AI judge prompts are specific and binary - [ ] Arguments match expected types (string vs number) - [ ] API keys and permissions are valid - [ ] No rate limits or quota issues @@ -1119,7 +1077,7 @@ Build up complexity gradually: **Performance:** - Exit early on critical failures - - Keep conversations focused (5-10 turns) + - Keep only the context needed for each decision - Batch related tests together **Maintenance:** diff --git a/fern/observability/evals-quickstart.mdx b/fern/observability/evals-quickstart.mdx index d100ea9b6..2617fca0f 100644 --- a/fern/observability/evals-quickstart.mdx +++ b/fern/observability/evals-quickstart.mdx @@ -6,7 +6,9 @@ slug: observability/evals-quickstart ## Overview -This quickstart guide will help you set up automated testing for your AI assistants and squads. In just a few minutes, you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production. +This quickstart shows how to check an assistant or squad's next decision using a mock conversation. You'll define the context, choose a judge, and inspect the result. + +Evals run at the text and model layer. They don't test speech recognition, audio quality, or turn-taking. Use [Simulations](/test/simulations-best-practices) for complete conversations, then controlled real calls for the phone path and live integrations. For operator-focused guidance on choosing checkpoints and writing durable checks, see [test decisions with Evals](/test/evals-best-practices). @@ -36,13 +38,15 @@ Evals help you maintain quality and catch issues early: ### What you'll build -An evaluation suite for an appointment booking assistant that tests: +Focused Evals for an appointment-booking assistant that test: -- Greeting and initial response validation -- Tool call execution with specific arguments +- Asking for missing information +- Tool call requests with specific arguments - Response pattern matching with regex - Semantic validation using AI judges -- Multi-turn conversation flows +- Decisions at selected conversation checkpoints + +These are separate saved Evals, not a native Eval suite. Group and run them yourself or use [custom automation](/observability/evals-advanced#cicd-integration). ## Prerequisites @@ -62,7 +66,7 @@ An evaluation suite for an appointment booking assistant that tests: ## Step 1: Create your first evaluation -Define a mock conversation to test your assistant's greeting behavior. +Start with a booking assistant whose policy requires a timezone before booking. This example checks whether it asks for the missing timezone. Include prior messages establishing any other prerequisites, such as verified identity, so asking for the timezone is the appropriate next decision. Choose an equivalent decision from your own assistant's requirements if it doesn't handle bookings. @@ -74,26 +78,26 @@ Define a mock conversation to test your assistant's greeting behavior. - 1. **Name**: Enter "Greeting Test" - 2. **Description**: Add "Verify assistant greets users appropriately" + 1. **Name**: Enter "Asks for missing timezone" + 2. **Description**: Add "Ask for the timezone before booking" 3. **Type**: Automatically set to "chat.mockConversation" 1. Click **Add Message** 2. Select **User** message type - 3. Enter content: "Hello" + 3. Enter content: "I want to book for January 20, 2027 at 2pm." 4. Click **Add Message** again 5. Select **Assistant** message type 6. Click **Enable Evaluation** toggle - 7. Select **Exact Match** as judge type - 8. Enter expected content: "Hello! How can I help you today?" + 7. Select **AI Judge**, then choose OpenAI and `gpt-4.1` + 8. Enter: `Context: {{messages}}. Evaluate the last assistant message. Pass only if it asks the caller to specify their timezone before proceeding. Fail if it assumes a timezone or proceeds to book. Respond only with pass or fail.` 9. Click **Save Evaluation** - Your evaluation is now saved. You can run it against any assistant or squad. + Run this Eval against assistants or squads with the same booking requirement. Before relying on the judge, check that it accepts a response asking for the timezone and rejects one that chooses a timezone without asking. @@ -104,19 +108,26 @@ curl -X POST "https://api.vapi.ai/eval" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ - "name": "Greeting Test", - "description": "Verify assistant greets users appropriately", + "name": "Asks for missing timezone", + "description": "Ask for the timezone before booking", "type": "chat.mockConversation", "messages": [ { "role": "user", - "content": "Hello" + "content": "I want to book for January 20, 2027 at 2pm." }, { "role": "assistant", "judgePlan": { - "type": "exact", - "content": "Hello! How can I help you today?" + "type": "ai", + "model": { + "provider": "openai", + "model": "gpt-4.1", + "messages": [{ + "role": "system", + "content": "Context: {{messages}}. Evaluate the last assistant message. Pass only if it asks the caller to specify their timezone before proceeding. Fail if it assumes a timezone or proceeds to book. Respond only with pass or fail." + }] + } } } ] @@ -130,8 +141,8 @@ curl -X POST "https://api.vapi.ai/eval" \ "id": "550e8400-e29b-41d4-a716-446655440000", "orgId": "org-123", "type": "chat.mockConversation", - "name": "Greeting Test", - "description": "Verify assistant greets users appropriately", + "name": "Asks for missing timezone", + "description": "Ask for the timezone before booking", "messages": [...], "createdAt": "2024-01-15T09:30:00Z", "updatedAt": "2024-01-15T09:30:00Z" @@ -159,7 +170,7 @@ Execute the evaluation against your assistant or squad. 1. Navigate to **Evals** in the sidebar - 2. Click on "Greeting Test" from your evaluations list + 2. Click on "Asks for missing timezone" from your evaluations list @@ -188,6 +199,7 @@ curl -X POST "https://api.vapi.ai/eval/run" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ + "type": "eval", "evalId": "550e8400-e29b-41d4-a716-446655440000", "target": { "type": "assistant", @@ -218,6 +230,8 @@ curl -X GET "https://api.vapi.ai/eval/run/eval-run-123" \ For complete API details, see [Create Eval Run](/api-reference/eval/run) and [Get Eval Run](/api-reference/eval/get-run). +The POST response confirms submission, not success. Poll the GET endpoint until `status` is `ended`, with a deadline. Then check `endedReason` and the results. See the [complete CI example](/observability/evals-advanced#cicd-integration). + @@ -246,11 +260,11 @@ When all checks pass, you'll see: "messages": [ { "role": "user", - "content": "Hello" + "content": "I want to book for January 20, 2027 at 2pm." }, { "role": "assistant", - "content": "Hello! How can I help you today?", + "content": "What timezone should I use for your appointment?", "judge": { "status": "pass" } @@ -282,14 +296,14 @@ When validation fails, you'll see details: "messages": [ { "role": "user", - "content": "Hello" + "content": "I want to book for January 20, 2027 at 2pm." }, { "role": "assistant", - "content": "Hi there! What can I do for you?", + "content": "I will use Pacific time and book that now.", "judge": { "status": "fail", - "failureReason": "Expected exact match: 'Hello! How can I help you today?' but got: 'Hi there! What can I do for you?'" + "failureReason": "The assistant assumed a timezone instead of asking." } } ] @@ -316,23 +330,23 @@ Validate that your assistant calls functions with correct arguments. ### Basic tool call validation -Test appointment booking with exact argument matching: +Test appointment booking with exact argument matching. This fixture uses an explicit date and an assistant configured to use `America/Los_Angeles` for bookings. Its `bookAppointment` tool accepts `date` and `time`. Supply any other required identity, availability, or policy context for your assistant before the checkpoint. Use a date your test environment accepts and keep the request and expected arguments in sync. 1. Create new evaluation: "Appointment Booking Test" - 2. Add user message: "Book me an appointment for next Monday at 2pm" + 2. Add user message: "Book me an appointment for January 20, 2027 at 2pm Pacific time" 3. Add assistant message with evaluation enabled 4. Select **Exact Match** judge type 5. Click **Add Tool Call** 6. Enter function name: "bookAppointment" 7. Add arguments: - - `date`: "2025-01-20" + - `date`: "2027-01-20" - `time`: "14:00" 8. Add tool response message: - Type: **Tool** - Content: `{"status": "success", "confirmationId": "APT-12345"}` - 9. Add final assistant message to verify confirmation + 9. Add a final assistant message with the AI judge below to check how it reports the mocked success 10. Save evaluation @@ -347,7 +361,7 @@ curl -X POST "https://api.vapi.ai/eval" \ "messages": [ { "role": "user", - "content": "Book me an appointment for next Monday at 2pm" + "content": "Book me an appointment for January 20, 2027 at 2pm Pacific time" }, { "role": "assistant", @@ -356,7 +370,7 @@ curl -X POST "https://api.vapi.ai/eval" \ "toolCalls": [{ "name": "bookAppointment", "arguments": { - "date": "2025-01-20", + "date": "2027-01-20", "time": "14:00" } }] @@ -369,8 +383,15 @@ curl -X POST "https://api.vapi.ai/eval" \ { "role": "assistant", "judgePlan": { - "type": "regex", - "content": ".*confirmed.*APT-12345.*" + "type": "ai", + "model": { + "provider": "openai", + "model": "gpt-4.1", + "messages": [{ + "role": "system", + "content": "Context: {{messages}}. Pass only if the last assistant message accurately reports the successful tool result and shares confirmation ID APT-12345. Otherwise fail. Respond only with pass or fail." + }] + } } } ] @@ -384,6 +405,8 @@ For API details, see [Create Eval](/api-reference/eval/eval-controller-create). ### Tool call validation modes +The mock response doesn't create an appointment. Verify external calendar state separately using a sandbox integration. Add a paired Eval where the tool returns an error and the assistant must not claim success, plus a case where missing information should prevent the tool call. + **Exact match - Full validation:** ```json @@ -394,7 +417,7 @@ For API details, see [Create Eval](/api-reference/eval/eval-controller-create). { "name": "bookAppointment", "arguments": { - "date": "2025-01-20", + "date": "2027-01-20", "time": "14:00" } } @@ -420,7 +443,7 @@ Validates both function name AND all argument values exactly. } ``` -Validates only that the function was called (arguments can vary). +Validates only that the function was called (arguments can vary). Use this only when the name alone is the requirement. Keep argument checks when a wrong date, account ID, or other value would make the action incorrect. **Multiple tool calls:** @@ -431,11 +454,11 @@ Validates only that the function was called (arguments can vary). "toolCalls": [ { "name": "checkAvailability", - "arguments": { "date": "2025-01-20" } + "arguments": { "date": "2027-01-20" } }, { "name": "bookAppointment", - "arguments": { "date": "2025-01-20", "time": "14:00" } + "arguments": { "date": "2027-01-20", "time": "14:00" } } ] } @@ -479,7 +502,7 @@ Matches: "Hello, I can help...", "Hi I'll help...", "Hey let me help..." } ``` -Matches any confirmation message with appointment ID format. +This checks a format, not booking success. It also matches "Your appointment is not confirmed. Failed request reference APT-12345." Use an AI judge to check meaning and tool checks to validate the requested action. **Date patterns:** @@ -607,7 +630,7 @@ Output format: respond with exactly one word: pass or fail - `{{messages[-1]}}` - The last assistant message only -### Example: Evaluate helpfulness and tone +### Example: Check for a clarifying question @@ -625,7 +648,7 @@ curl -X POST "https://api.vapi.ai/eval" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ - "name": "Helpfulness Test", + "name": "Asks what account help is needed", "type": "chat.mockConversation", "messages": [ { @@ -641,7 +664,7 @@ curl -X POST "https://api.vapi.ai/eval" \ "model": "gpt-4o", "messages": [{ "role": "system", - "content": "You are an LLM-Judge. Evaluate ONLY the last assistant message: {{messages[-1]}}.\n\nInclude context: {{messages}}\n\nDecision rule:\n- PASS if ALL pass criteria are met AND NO fail criteria are triggered.\n- Otherwise FAIL.\n\nPass criteria:\n- Response acknowledges the user request\n- Response offers specific help or next steps\n- Tone is professional and friendly\n\nFail criteria (any triggers FAIL):\n- Response is rude or dismissive\n- Response ignores the user request\n- Response provides no actionable information\n\nOutput format: respond with exactly one word: pass or fail" + "content": "Context: {{messages}}. Evaluate only the last assistant message: {{messages[-1]}}. Pass if it asks what account issue the caller needs help with. Fail if it assumes a specific issue without clarification. Respond only with pass or fail." }] } } @@ -756,6 +779,8 @@ Provide fallback responses to continue testing even when validation fails: ### Example: Multi-step with exit control +This example assumes the assistant is required to say "This call may be recorded." exactly. Don't exact-match ordinary conversational wording. The later override supplies context for debugging; it doesn't turn a failed checkpoint into a pass or prove that the booking happened. + 1. Create evaluation with multiple conversation turns @@ -783,7 +808,7 @@ curl -X POST "https://api.vapi.ai/eval" \ "role": "assistant", "judgePlan": { "type": "exact", - "content": "I can help you book an appointment." + "content": "This call may be recorded." }, "continuePlan": { "exitOnFailureEnabled": true @@ -791,7 +816,7 @@ curl -X POST "https://api.vapi.ai/eval" \ }, { "role": "user", - "content": "Monday at 2pm" + "content": "January 20, 2027 at 2pm Pacific time" }, { "role": "assistant", @@ -801,10 +826,10 @@ curl -X POST "https://api.vapi.ai/eval" \ }, "continuePlan": { "exitOnFailureEnabled": false, - "contentOverride": "Booking confirmed for Monday at 2pm.", + "contentOverride": "Booking confirmed for January 20, 2027 at 2pm Pacific time.", "toolCallsOverride": [{ "name": "bookAppointment", - "arguments": {"date": "2025-01-20", "time": "14:00"} + "arguments": {"date": "2027-01-20", "time": "14:00"} }] } } @@ -824,25 +849,27 @@ curl -X POST "https://api.vapi.ai/eval" \ Validate multi-turn interactions that simulate real user conversations. -### Complete booking flow example +### Fixed-path booking checkpoints + +This example checks selected decisions along a supplied path. It isn't an end-to-end booking test. Use a Simulation to explore different paths to the outcome. Use the tool and timezone setup from Step 4, and supply the caller's test email in the mock context before expecting `sendEmail`. - Create a comprehensive test: + Check each selected decision: 1. **Turn 1 - Initial request:** - User: "I need to schedule an appointment" - Assistant evaluation: AI judge checking acknowledgment 2. **Turn 2 - Provide details:** - - User: "Next Monday at 2pm" + - User: "January 20, 2027 at 2pm Pacific time" - Assistant evaluation: Exact match on tool call `bookAppointment` 3. **Turn 3 - Tool response:** - Tool: `{"status": "success", "confirmationId": "APT-12345"}` 4. **Turn 4 - Confirmation:** - - Assistant evaluation: Regex matching confirmation with ID + - Assistant evaluation: AI judge checking that the reply accurately reports the tool result and confirmation ID 5. **Turn 5 - Follow-up:** - User: "Can I get that via email?" @@ -855,8 +882,8 @@ curl -X POST "https://api.vapi.ai/eval" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ - "name": "Complete Booking Flow", - "description": "Test full appointment booking conversation", + "name": "Booking checkpoints", + "description": "Check decisions along a fixed booking path", "type": "chat.mockConversation", "messages": [ { @@ -879,7 +906,7 @@ curl -X POST "https://api.vapi.ai/eval" \ }, { "role": "user", - "content": "Next Monday at 2pm" + "content": "January 20, 2027 at 2pm Pacific time" }, { "role": "assistant", @@ -888,7 +915,7 @@ curl -X POST "https://api.vapi.ai/eval" \ "toolCalls": [{ "name": "bookAppointment", "arguments": { - "date": "2025-01-20", + "date": "2027-01-20", "time": "14:00" } }] @@ -901,8 +928,15 @@ curl -X POST "https://api.vapi.ai/eval" \ { "role": "assistant", "judgePlan": { - "type": "regex", - "content": ".*confirmed.*APT-12345.*" + "type": "ai", + "model": { + "provider": "openai", + "model": "gpt-4.1", + "messages": [{ + "role": "system", + "content": "Context: {{messages}}. Pass only if the last assistant message accurately reports the successful tool result and shares confirmation ID APT-12345. Otherwise fail. Respond only with pass or fail." + }] + } } }, { @@ -974,15 +1008,12 @@ Inject system prompts mid-conversation to test dynamic behavior changes: ``` - **Multi-turn testing tips:** - Keep conversations focused (5-10 turns for most - tests) - Use exit-on-failure for early turns to save time - Test one primary - flow per evaluation - Mix judge types (exact, regex, AI) for comprehensive - validation - Include tool responses to simulate real interactions + Keep only the context needed for each decision and choose the simplest judge that proves its requirement. Include relevant tool results. Use exit-on-failure when later checkpoints depend on an earlier one passing. Use a Simulation for the complete conversation outcome. ## Step 9: Manage evaluations -List, update, and organize your evaluation suite. +List and update individual Evals. Organize groups in your own tracker or automation. ### List all evaluations @@ -1184,7 +1215,7 @@ For API details, see [List Eval Runs](/api-reference/eval/list-runs). "messages": [ { "role": "user", - "content": "Book an appointment for Monday at 2pm" + "content": "Book an appointment for January 20, 2027 at 2pm Pacific time" }, { "role": "assistant", @@ -1193,7 +1224,7 @@ For API details, see [List Eval Runs](/api-reference/eval/list-runs). { "name": "bookAppointment", "arguments": { - "date": "2025-01-20", + "date": "2027-01-20", "time": "2:00 PM" } } @@ -1224,56 +1255,7 @@ For API details, see [List Eval Runs](/api-reference/eval/list-runs). ### Multiple validation types in one eval -Combine exact, regex, and AI judges for comprehensive testing: - -```json -{ - "messages": [ - { - "role": "user", - "content": "Hello" - }, - { - "role": "assistant", - "judgePlan": { - "type": "exact", - "content": "Hello! How can I help you?" - } - }, - { - "role": "user", - "content": "Book appointment for Monday" - }, - { - "role": "assistant", - "judgePlan": { - "type": "regex", - "content": ".*(Monday|next week).*" - } - }, - { - "role": "user", - "content": "Thanks for your help" - }, - { - "role": "assistant", - "judgePlan": { - "type": "ai", - "model": { - "provider": "openai", - "model": "gpt-4o", - "messages": [ - { - "role": "system", - "content": "PASS if response is polite and acknowledges thanks. Output: pass or fail" - } - ] - } - } - } - ] -} -``` +Choose a judge for each checkpoint's requirement, not to include every judge type in one test. Use exact matching for required tool arguments or mandated wording, regex for a stable format, and an AI judge for a decision's meaning. Split independent questions into separate checks so a failure is easy to explain. Use a Simulation when you want to test a complete journey through different valid paths. ### Test squad handoffs diff --git a/fern/observability/monitoring-quickstart.mdx b/fern/observability/monitoring-quickstart.mdx index ccfefd620..12fedc3db 100644 --- a/fern/observability/monitoring-quickstart.mdx +++ b/fern/observability/monitoring-quickstart.mdx @@ -6,7 +6,9 @@ slug: observability/monitoring-quickstart ## Overview -Monitoring lets you automatically track quality and detect issues across your voice AI agents. Instead of manually reviewing calls, you define monitors that continuously evaluate your call data against thresholds and alert you when something goes wrong. +Monitoring lets you automatically track quality and detect issues across your voice AI agents. Alongside regular call review, define monitors that evaluate your call data against thresholds and alert you when those thresholds are exceeded. + +Monitors only detect the conditions you configure. Review a sample of unflagged calls too, and turn confirmed production failures into [regression tests](/test/run-and-maintain-tests#turn-production-issues-into-regression-tests). ### What is monitoring? diff --git a/fern/observability/scorecard-quickstart.mdx b/fern/observability/scorecard-quickstart.mdx index b72ff67d8..712a4e85e 100644 --- a/fern/observability/scorecard-quickstart.mdx +++ b/fern/observability/scorecard-quickstart.mdx @@ -10,7 +10,7 @@ This quickstart shows how to create a scorecard that automatically grades calls ### What are scorecards? -Scorecards compute objective quality metrics from the call's structured outputs after the call ends: +Scorecards apply scoring rules to the call's structured outputs after the call ends: 1. Evaluate against metrics - Compare structured output values to conditions you define 2. Allocate points - Grant points when conditions are met, per metric @@ -32,6 +32,8 @@ Scorecards compute objective quality metrics from the call's structured outputs - Grade calls consistently and automatically providing a quick overview of the call's quality - Power dashboards and automated workflows with numeric scores +The scoring rules are consistent, but AI-generated structured outputs can be wrong. Before relying on a score, compare its inputs with calls that reviewers agree should pass or fail. Investigate disagreements and sample both high- and low-scoring calls. Keep critical policy or business failures visible separately; a high total score doesn't cancel them out. See [reviewing automated results](/test/run-and-maintain-tests#review-more-than-the-pass-or-fail-label). + ## What you'll build A scorecard that: diff --git a/fern/observability/simulations-manage.mdx b/fern/observability/simulations-manage.mdx index c72a84300..1dfca295b 100644 --- a/fern/observability/simulations-manage.mdx +++ b/fern/observability/simulations-manage.mdx @@ -106,7 +106,7 @@ Build coverage gradually as the assistant or squad changes. ### Smoke tests -Start with the core path that must always work. Use a direct intent, one or two required Boolean evaluations, and a single iteration. Chat mode is useful for validating conversation and tool logic quickly. +Start with a core path, a direct intent, and one or two required Boolean evaluations. One iteration is useful while setting up or debugging the test, not as evidence of reliability. Repeat critical release checks based on risk and past variation. Chat mode gives quick feedback on conversation and tool logic. ### Regression tests @@ -114,7 +114,7 @@ Create a regression test when you fix a defect or find an unexpected response. R ### Edge-case tests -Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. +Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. ### Design evaluations @@ -125,9 +125,11 @@ Keep each evaluation focused on one observable outcome. Use a descriptive name, | Testing goal | Mode | Why | | --- | --- | --- | | Iterate on prompts, tools, and conversation logic | Chat (`vapi.webchat`) | Runs without audio processing, so it is faster and costs less. | -| Validate speech recognition, voice output, or interruptions | Voice (`vapi.websocket`) | Exercises the complete audio path. | +| Investigate speech recognition, voice output, or interruptions | Voice (`vapi.websocket`) | Exercises synthetic audio. Review the recording, not just the pass/fail label. | | Receive call-specific webhook data | Voice (`vapi.websocket`) | Start and end webhooks fire in both modes, but chat payloads omit call-specific fields. | -| Perform final pre-launch validation | Voice (`vapi.websocket`) | Tests the assistant or squad in an end-to-end voice conversation. | +| Check representative voice journeys before launch | Voice (`vapi.websocket`) | Adds voice coverage. Also make controlled calls through the real phone path and sandbox integrations. | + +Review failures and sample passing runs. Compare automated judgments with human-reviewed examples, and listen to voice recordings when available. Follow [run and maintain tests](/test/run-and-maintain-tests) for release checks and failure triage. ## Maintain simulation coverage diff --git a/fern/observability/simulations-quickstart.mdx b/fern/observability/simulations-quickstart.mdx index 99f9e3879..1056c07e8 100644 --- a/fern/observability/simulations-quickstart.mdx +++ b/fern/observability/simulations-quickstart.mdx @@ -1,7 +1,7 @@ --- title: Simulations quickstart subtitle: Run your first simulation against an assistant or squad in minutes. -description: "Create a simulation suite that asks an AI tester to book an appointment with Riley, then evaluate whether it was booked for the requested date and time." +description: "Create a simulation suite for Riley and check whether the conversation reports the requested appointment details." slug: observability/simulations-quickstart --- @@ -9,7 +9,7 @@ In about 10 minutes, you'll run an AI tester through an appointment-booking conv ## What you'll build -A reusable **simulation suite** that asks an AI tester to book an appointment with Riley, Vapi's appointment-scheduler template assistant. A Boolean evaluation checks whether Riley books the appointment for next Monday at 2:00 PM. +A reusable **simulation suite** that asks an AI tester to book an appointment with Riley, Vapi's appointment-scheduler template assistant. The first criterion checks whether Riley reports the requested date and time with a confirmation number. This is a conversation-only demonstration, not proof that an appointment exists in a calendar. ## Prerequisites @@ -17,6 +17,10 @@ A reusable **simulation suite** that asks an AI tester to book an appointment wi You create Riley and the structured output during this quickstart. + + Simulations run unmocked tools for real. Before running, inspect Riley's tools and use sandbox integrations or [mock every tool](/observability/simulations-advanced#mock-tool-responses) that could create a booking, send a message, or affect a real customer. + + ## Build and run a simulation @@ -45,8 +49,9 @@ You create Riley and the structured output during this quickstart. The suite opens with its first simulation. In the **Scenario** tab, enter `Book an appointment` as the **Scenario name**. Paste the following text into **Intent**: ```text - You are calling to book an appointment for next Monday at 2pm. - Confirm your identity when asked and provide any required information. + You are Alex Taylor, calling to book an appointment for January 20, 2027 + at 2pm in America/Los_Angeles. Your test email is alex@example.com. + Provide these details when asked. Don't invent other account information. End the call once you receive a confirmation number. ``` @@ -80,9 +85,9 @@ You create Riley and the structured output during this quickstart. | Setting | Value | | -- | -- | - | **Name** | `appointment_booked` | + | **Name** | `appointment_details_reported` | | **Type** | Boolean | - | **Description** | Return `true` if Riley books the appointment for next Monday at 2:00 PM. | + | **Description** | Return `true` only if Riley tells the caller the appointment is confirmed for January 20, 2027 at 2pm Pacific time and provides a confirmation number. Otherwise return `false`. This checks the reported details, not calendar state. | | **Comparator** | `equals (=)` | | **Expected value** | `true` | @@ -97,10 +102,10 @@ You create Riley and the structured output during this quickstart. - Runs faster and costs less because it skips audio evaluation. + Runs without speech generation or transcription, so it is faster and costs less. - Tests the complete audio experience at about twice the cost. + Exercises speech, transcription, and turn-taking with a synthetic caller. Listen to the recording to assess the audio experience. @@ -121,7 +126,7 @@ You create Riley and the structured output during this quickstart. The header shows Riley, the mode, run date, and overall result (`Passed` or a count such as `1/1 failed`). The list on the left shows one result for each simulation and iteration. Select a result to review it: - The **Success criteria** panel appears next to the **Transcript**. The `appointment_booked` evaluation passes when Riley books the appointment for next Monday at 2:00 PM. The transcript lists the conversation turn by turn between the AI tester and Riley. + The **Success criteria** panel appears next to the **Transcript**. Review whether `appointment_details_reported` agrees with what Riley actually said. The transcript lists the conversation turn by turn between the AI tester and Riley. For voice runs, also listen to the recording when available; the criterion doesn't grade pronunciation, pauses, or interruptions. Configure variables, tool mocks, webhooks, and structured outputs. diff --git a/fern/workflows/legacy-migration.mdx b/fern/workflows/legacy-migration.mdx index 914ddecb9..b0e453e47 100644 --- a/fern/workflows/legacy-migration.mdx +++ b/fern/workflows/legacy-migration.mdx @@ -72,7 +72,7 @@ Before you go live, verify: - Every handoff condition triggers at the right time - Variable extraction passes the right data between assistants - Tools, API requests, and transfers are attached to the correct assistant -- Test calls cover **all** conversation paths, including edge cases +- Tests cover critical decisions, important customer outcomes, and likely failure paths If anything looks off, fix it manually using the steps below. @@ -156,11 +156,14 @@ If anything looks off, fix it manually using the steps below. ### Step 6 — Test - Use the built-in calling feature to test all conversation paths before going live. Pay particular attention to: + Build coverage around the risks in the migrated Squad, not every possible conversation: - - Handoff conditions triggering at the right time - - Variables passing correctly between assistants - - Edge cases (confused users, unexpected inputs, human escalation paths) + - Use Evals to check handoff decisions and required tool arguments, including when a handoff must not happen. + - Use Simulations to check complete journeys, recovery from tool errors, and whether downstream assistants receive the information they need. + - Repeat critical journeys and investigate inconsistent results before launch. + - Make controlled calls to verify audio, real transfers, and sandbox integrations. Don't let tests affect real customers or production records. + + See [plan test coverage](/test/plan-test-coverage) and [release checks](/test/run-and-maintain-tests). ## Useful resources