Rewrite voice agent testing best practices - #1193
Conversation
|
🌿 Preview your docs: https://vapi-preview-01a06ee1-3ed4-76ed-a81a-6836c33a6d6f.docs.buildwithfern.com |
Lightsage docs evalsResult: passed Average score: 100/100
|
627f6e4 to
d628495
Compare
| | **Core outcome** | The caller completes the goal and receives an accurate confirmation or next step. | Simulation | | ||
| | **Critical decisions** | The agent asks the required question, chooses the right tool or route, and uses the right information. | Eval | | ||
| | **Both directions** | The action happens when it should and does not happen when it should not. | Evals | | ||
| | **Missing or unclear information** | The agent asks for what it needs instead of guessing. | Eval, then Simulation | |
There was a problem hiding this comment.
mega nit:
| | **Missing or unclear information** | The agent asks for what it needs instead of guessing. | Eval, then Simulation | | |
| | **Missing or unclear information** | The agent asks for missing data instead of hallucinating them. | Eval, then Simulation | |
There was a problem hiding this comment.
Kept 'asks for what it needs instead of guessing' in fern/test/plan-test-coverage.mdx. It describes the expected behavior in plain language for support operators and product managers; 'hallucinating' adds AI terminology without making the requirement more specific. Leaving this open for your follow-up.
chris-garber-vapi
left a comment
chris-garber-vapi
left a comment
There was a problem hiding this comment.
LGTM, just a few tune up comments!
| - Voice tests require more time to execute compared to chat tests. | ||
| - Each test consumes calling minutes from your account. | ||
| - Maximum call duration is limited to 15 minutes per test. | ||
| For example, use an **Eval** to check that an agent calls the rescheduling tool only after it verifies the caller. Use a **Simulation** to check that a caller can reschedule, hear the correct new time, and finish the conversation successfully. The Simulation should pass even if the agent asks the questions in a different valid order. |
There was a problem hiding this comment.
For example, use an Eval to check that an agent calls the rescheduling tool only after verifying the caller. Use a Simulation to check that a caller can reschedule, hear the correct new time, and finish successfully - and it should pass even if the agent asks the questions in a different valid order.
There was a problem hiding this comment.
This example is already present in fern/test/voice-testing.mdx under 'Use Evals for decisions and Simulations for outcomes': identity verification before rescheduling, followed by a Simulation that accepts different valid question orders. Retained it without adding a duplicate. Leaving this open in case you intended a different change.
|
🌿 Preview your docs: https://vapi-preview-01a0823a-e6f9-76d7-bd24-4c2840ab56ba.docs.buildwithfern.com |
|
|
||
| ## Prioritize by risk | ||
|
|
||
| You do not need hundreds of tests to get started. Test these areas first: |
|
Follow-up on the Lightsage evaluation report: the failed Codex run at The push of |
|
🌿 Preview your docs: https://vapi-preview-01a0835b-16eb-7739-ae1f-a6b5ebd69632.docs.buildwithfern.com |
|
🌿 Preview your docs: https://vapi-preview-01a083a7-66e4-725f-bf47-15e9c4b71620.docs.buildwithfern.com |
4 participants
Description
Testing Steps
fern docs devor navigate to preview deploymentValidation for review follow-up
5094ba5:fern checkpassed with 0 errors and 12 existing API warnings. Rendered all six edited pages in an isolated local Fern preview with the repository stylesheet, and visually checked the coverage checklist. No executable examples were added; the unsupported Eval-group JSON was removed. The full hosted preview is validated separately by CI.