Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions fern/apis/api/openapi-overrides.yml
Original file line number Diff line number Diff line change
Expand Up @@ -998,6 +998,16 @@ components:
A record of an eval execution, including its target, status, results,
costs, completion details, and lifecycle timestamps.
properties:
results:
description: >-
Results for this individual Eval. Check them after status is ended.
An Eval that finishes normally contains one result; it passes only
when all judged checkpoints pass. Grouping multiple Evals requires your
own orchestration, not an Eval suite.
cost:
description: The cost of this Eval run in USD.
costs:
description: The cost breakdown for this Eval run.
id:
description: The unique identifier for the eval run.
orgId:
Expand Down
11 changes: 10 additions & 1 deletion fern/assistants/examples/appointment-scheduling.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,16 @@ Use the Google Calendar integration for availability and booking, or your own AP

## 5. Test and validate

Use sandbox calendar records and test identities. Don't create, change, or cancel real customer appointments during testing.

| Check | Coverage |
| --- | --- |
| Evals | Correct booking arguments when all required details are known; ask when timezone is missing; don't confirm after a tool error. |
| Simulations | A complete booking, an unavailable slot with an alternative, and rescheduling without losing the original appointment if the change fails. |
| Controlled calls | Listen for date and time clarity, and verify the intended changes actually exist in the sandbox calendar. |

Repeat critical scenarios before release. A tool mock checks behavior under a supplied result, not the calendar integration. See [plan test coverage](/test/plan-test-coverage).

<Steps>
<Step title="Attach a phone number">
Create a phone number and assign your assistant. See [Phone calls quickstart](/quickstart/phone).
Expand All @@ -274,4 +284,3 @@ Use the Google Calendar integration for availability and booking, or your own AP
- **Tools**: [Google Calendar](/tools/google-calendar), [Custom Tools](/tools/custom-tools)
- **Structured outputs**: [Extract structured data](/assistants/structured-outputs-quickstart)
- **Multichannel**: [Web integration](/quickstart/web)

10 changes: 10 additions & 0 deletions fern/assistants/examples/support-escalation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -705,6 +705,16 @@ Always be professional and efficient in your support."""

## 4. Test Your Support Escalation System

Use dedicated test accounts and transfer destinations staffed by your team. The scripts below create calls for manual testing; they don't simulate callers or assert that escalation worked.

| Check | Coverage |
| --- | --- |
| Evals | Escalate when policy requires it, avoid unnecessary escalation, and request the correct destination with the required context. |
| Simulations | Follow an unresolved issue through escalation, including a failed or unavailable destination and a safe next step. |
| Controlled calls | Verify the real transfer connects, the recipient receives the needed context, and the audio remains usable. |

Repeat critical checks and turn routing failures into regression tests. See [test decisions with Evals](/test/evals-best-practices) and [test outcomes with Simulations](/test/simulations-best-practices).

<Tabs>
<Tab title="Dashboard">
<Steps>
Expand Down
4 changes: 3 additions & 1 deletion fern/assistants/structured-outputs-examples.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1215,6 +1215,8 @@ Always test your structured outputs with these scenarios:
4. **Edge cases** - Boundary values, special characters
5. **Real conversations** - Actual call recordings or transcripts

Valid JSON doesn't guarantee a correct extraction or judgment. Compare each output with expected values from human-reviewed examples, including known successes and failures. For outcome judgments, record what evidence is available: a reported booking isn't proof of a calendar update. Use synthetic values or remove personal information from reused calls. Follow [judge calibration and result review](/test/run-and-maintain-tests#review-more-than-the-pass-or-fail-label).

### Monitoring checklist

Track these metrics for production deployments:
Expand Down Expand Up @@ -1291,4 +1293,4 @@ When retrieving call data via API:
- [Structured outputs overview](/assistants/structured-outputs) - Main documentation
- [Quickstart guide](/assistants/structured-outputs-quickstart) - Get started quickly
- [API reference](/api-reference#structured-output) - Complete API documentation
- [JSON Schema specification](https://json-schema.org/) - JSON Schema standard
- [JSON Schema specification](https://json-schema.org/) - JSON Schema standard
1 change: 1 addition & 0 deletions fern/assistants/structured-outputs-quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -912,6 +912,7 @@ Common validation patterns for reliable extraction:
- Use enums for categorical data to ensure consistency
- Add descriptions to help the AI understand context
- Test with real conversations before production use
- Compare extracted values and judgments with human-reviewed examples. A valid schema checks the output's shape, not whether its values are true.
- Monitor extraction success rates and iterate on schemas
</Note>

Expand Down
8 changes: 8 additions & 0 deletions fern/customization/multilingual.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,14 @@ Configure greeting messages that work across multiple languages.

Validate your configuration with different languages and scenarios.

| Check | Coverage |
| --- | --- |
| Evals | Check the next response or routing decision in each supported language, including when the caller changes language. These checks don't test speech recognition. |
| Voice Simulations | Run representative customer journeys in each language and with mixed-language input. Repeat critical cases and listen to recordings. |
| Controlled calls | Ask native speakers to review understanding, pronunciation, and turn-taking on the real phone path. Synthetic voices aren't complete accent or noise coverage. |

Use [plan test coverage](/test/plan-test-coverage) to prioritize the languages and journeys your customers depend on. Review both failed and passing calls.

<Tabs>
<Tab title="Dashboard">
1. Use the **Test Assistant** feature in your dashboard
Expand Down
Loading
Loading