Cortex Agent Evaluation Template

A template for evaluating a Cortex Agent across four dimensions, not just one. It ships a working demo agent that reads a skill file from a named stage, a small deterministic dataset, and five evaluation configs you can run and read immediately.

Fork it, point it at your own workflow, and you have a repeatable scorecard instead of a vibe check.

Read AGENT.md before you build on this. It records the behaviors verified against a live account, including five settings worth checking explicitly because a misconfiguration still produces a working run.

CODE-EXAMPLES.md has every command, given twice: once for Cortex Code and once for CLI-driven agents such as Claude Code or Cursor.

New to Cortex Agent evaluations? Start with the illustrated explainer, Exhibit C: Cortex Agent Evaluations. It covers the Goal-Plan-Action framework, the metrics, and the trace behavior this template implements, as a tabbed page you can read in one sitting.

Why four dimensions

Answer correctness alone hides the failures that matter with an agentic workflow. An agent is non-deterministic in its route, not just its wording, so you have to score the route as well as the destination.

Dimension	Question it answers	How
Answer correctness	Did it get the right answer?	Built-in answer_correctness
Logical consistency	Does the reasoning follow from the evidence?	Built-in logical_consistency
Tool selection	Did it call the right tools, and only those?	Custom LLM-as-judge over trace fields
Tool execution	Given the right tool, did it call it correctly?	Custom LLM-as-judge over trace fields

The pattern across all four is what tells you where to look:

• Answer high, tool selection low: right answers via an unintended route. Fragile on untested questions.
• Tool selection high, tool execution low: right tool, wrong arguments. Fix the tool description's input guidance, not the orchestration instructions.
• Both tool dimensions high, answer low: right evidence, misreported. Fix response instructions or the semantic view's AI_SQL_GENERATION.
• Everything high except the rows expecting no tool call: the agent cannot say "I need more information". Only tool selection catches this, and it is the most common failure here.

All four dimensions have built-in metrics: answer_correctness, logical_consistency, tool_selection_accuracy (TSA) and tool_execution_accuracy (TEA). Use them. This template then adds two custom metrics for the workflow rules a generic metric cannot know about.

Custom judge prompts get trace fields the built-ins do not expose: {{tool_info}}, {{tool_name}}, {{tool_input}}, {{tool_output}}, {{span_name}}, {{status}}, {{error}}, {{duration}}, alongside {{input}}, {{output}} and {{ground_truth}}.

Reserved metric names

answer_correctness, logical_consistency, tool_selection_accuracy and tool_execution_accuracy are reserved built-in names. If you define a custom metric using one of them, the built-in takes precedence and your prompt is not used. This does not raise an error, so check which one ran.

To detect it, check the eval span: ai.observability.eval.metric_type is system for a built-in and custom for yours. That is why the custom metrics here are named workflow_tool_routing and workflow_tool_arguments.

What makes the ground truth different

The stock example carries only the expected final answer, which is why it can score answers but not routing. Each row here carries a ground_truth_output plus a ground_truth_invocations array and two custom keys (expected_class, trap).

For the full VARIANT example and per-key semantics, see AGENT.md section 5 and the header of models/evaluations/eval_dataset.sql.

ground_truth_invocations is the key the tool metrics read. The name and shape are exact, and an unrecognized key does not fail, so TSA/TEA would score against input you did not intend:

Key	Required	Read by
tool_name	yes	TSA and TEA. Use the name the agent sees; a Cortex Analyst tool also matches its semantic view name. Web search is the literal web_search.
tool_input	no	TEA only. The natural-language question you expect the agent to pass, not SQL you author.
tool_output	no	TEA only. Expected result rows, SQL description, or citations, in whatever form is clearest.

• TSA scores matched / max(expected, actual), ignoring order, so it penalizes too few calls, too many, and wrong ones.
• TEA semantically matches each expected entry to the closest real call. An expected tool that never ran scores 0.0, but extra tools cost nothing. That asymmetry is exactly why you run both.
• Omit tool_input and tool_output and TEA only confirms the tool ran.

expected_class and trap are custom keys. Every built-in ignores them, while custom metrics receive the whole VARIANT through {{ground_truth}} regardless of key. Keep output criteria in ground_truth_output and process criteria in your own keys.

Two of the ten rows carry "ground_truth_invocations": [], asserting that the correct behavior is to call nothing: an out-of-scope question, and a single-issue request with no id. An agent that reflexively calls a tool passes the answer evals and fails these. A third row ("Tell me about the open defects") is a related trap: it does expect a tool, but the wrong one is tempting.

Prerequisites

Requirement	Details
Snowflake account	Cortex Agents enabled
Role	CREATE SEMANTIC VIEW, CREATE AGENT, CREATE STAGE, CREATE FUNCTION on the target schema
Warehouse	XS is fine
External Access Integration	Only for dbt deps to reach hub.getdbt.com
Snowflake CLI	>= 3.21 for the env.yml flags (--env, --default-env)

One-time external access setup so dbt can reach the package hub, needs ACCOUNTADMIN. DDL in CODE-EXAMPLES.md.

Setup

Order matters; each step depends on the previous one.

All commands live in CODE-EXAMPLES.md, written twice: once for Cortex Code and once for CLI-driven agents such as Claude Code or Cursor. This section covers what each step does and what to watch for.

Step	What happens	Notes
1. Point it at your environment	Edit env.yml and, if you move the demo data, models/sources.yml.	dev uses a per-developer schema via CURRENT_USER(), so nobody overwrites anyone.
2. Create and load the demo data	sql/00_setup.sql, sql/01_load_demo_data.sql, sql/02_custom_tool.sql.	The loader is a stored procedure, not a dbt macro, so the rows live in one place. Idempotent.
3. Upload the skill	sql/03_skill_stage.sql creates the stage, uploads SKILL.md, and lists it back.	Needs PUT, so this one goes through the CLI either way. Do not skip the verification.
4. Build the models	snow dbt deploy, then snow dbt execute ... build.	Builds the staging views, semantic view, and eval_dataset.
5. Deploy the agent	run-operation deploy_workflow_agent, with {alter: true} for a zero-downtime update.	Then DESCRIBE AGENT to confirm skills attached.
6. Run the evaluations	create_eval_stage, upload the config, run_evaluation.	The macro does not upload the config. Re-upload after every edit.

Demo data is fixed in time

January to March 2026, because the ground truth asserts exact figures against it:

	Revenue (excl. cancelled)
West	$13,305.00
East	$8,655.00
Central	$6,855.00
Total	$28,815.00 across 21 of 24 orders

Change the data and you must update seeds/eval_ground_truth.csv, or answer correctness stops meaning anything.

One dataset per source table

The dataset: block in a config creates a dataset object. After the first successful run, comment it out or later runs fail with Dataset version ... already exists. The version name is a fixed constant, so there is one dataset per source table regardless of dataset_name. All five configs share it. Details in evaluations/AGENT.md.

The skills syntax, verified

The public CREATE AGENT reference and the Cortex Agents REST API reference do not document a skills field at all. The syntax below was established by testing against a live account.

skills is a top-level key, a sibling of models / instructions / tools / tool_resources. It is not nested under tool_resources:

skills:
  - name: triage
    source:
      type: STAGE
      path: "@DB.SCHEMA.SKILL_STAGE/skills/triage"

Verified	Result
skills in CREATE AGENT ... FROM SPECIFICATION	Works
skills in ALTER AGENT ... MODIFY LIVE VERSION SET SPECIFICATION	Works
description inside a skill entry	Rejected: agent spec is invalid: unrecognized field description
Multiple skills in one array	Works
Round-trip through DESCRIBE AGENT	Exact

Two things to verify

1. Path validation is lazy. An agent pointing at a nonexistent stage path creates successfully with no error. A typo gives you a clean deploy and an agent that quietly has no skill. The only defence is to verify separately, every time:

LS '@DB.SCHEMA.SKILL_STAGE' PATTERN='.*SKILL\.md';

2. The stage and the upload both have non-default requirements.

CREATE STAGE ... ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');  -- CSE default is unreadable by the agent
PUT ... AUTO_COMPRESS = FALSE;                            -- a SKILL.md.gz is not discovered

SKILL.md must sit at the root of its skill folder; Snowflake does not search subdirectories. The description comes from the file's YAML frontmatter, which is why the spec entry does not take one.

Semantic view syntax notes

Also established by testing, since the published grammar does not cover SAMPLE_VALUES:

• SAMPLE_VALUES takes no equals sign. SAMPLE_VALUES ('a','b') is correct; SAMPLE_VALUES = (...) is a syntax error.
• AI_SQL_GENERATION and AI_QUESTION_CATEGORIZATION take no equals sign.
• Verified queries are <name> AS ( QUESTION '...' SQL '...' ), with no equals signs and no NAME key.
• Within a dimension, fact, or metric: AS <expr> then WITH SYNONYMS then COMMENT then SAMPLE_VALUES then IS_ENUM.
• Clause order is enforced: TABLES -> RELATIONSHIPS -> FACTS -> DIMENSIONS -> METRICS -> COMMENT -> AI_* -> AI_VERIFIED_QUERIES. COMMENT must precede the AI_* clauses.
• dbt renders Jinja inside SQL comments. No comment in a semantic view model may contain Jinja braces, even commented out.

Reading the results

Scores are on a 0.0 to 1.0 scale. EXECUTE_AI_EVALUATION is asynchronous: START returns immediately, then the run moves through an invocation phase (the agent answers every row) into a computation phase (the judges score them).

Scores are not returned by the call. They land in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS, where the score and its metadata sit on separate records that share an eval_root_id.

For the status-polling call and the score-retrieval query, see AGENT.md section 8. Those queries live in one place so they cannot drift.

Two things to know before you read any number:

• Match the status exactly when polling. COMPLETED is a substring of INVOCATION_PARTIALLY_COMPLETED, so a substring test reports success on a stalled run.
• metric_type is the collision check. custom means your prompt ran; system means a built-in ran in its place. Always confirm this on a new custom metric.

The verified-query false negative

If a question exactly matches an AI_VERIFIED_QUERIES entry on the semantic view, Cortex Analyst takes a fast path: it runs the verified SQL directly without loading the semantic model, and the trace shows system_execute_sql / SqlExecution with verified_query_used: true and no Analyst tool span at all.

A naive custom rubric then reports "the expected Analyst tool was never called" and scores a perfectly correct answer 0.0. The built-in TSA handles this correctly, which is a good reason to run it alongside. Both custom rubrics in evaluations/ carry an explicit instruction to treat a verified-query SQL execution as the Analyst call. Keep it if you edit them.

Troubleshooting

Symptom	Cause
Object name WORKFLOW_AGENT format should be 'database.schema.object'	agent_name must be fully qualified, even though some examples show a bare name.
Dataset version SYSTEM_AI_OBS_CORTEX_AGENT_DATASET_VERSION_DO_NOT_DELETE already exists	The version name is a fixed constant, so you get one dataset per source table, whatever you set dataset_name to. Reuse the dataset by removing the dataset: block, or point at another table.
The DB is not set for the current session	Prefix the call with USE DATABASE, USE SCHEMA, USE WAREHOUSE.
Run parks at INVOCATION_PARTIALLY_COMPLETED	Some rows never finished invocation. CANCEL, then re-run. Check for errored spans before assuming a real agent failure.
Custom prompt appears to have no effect	Metric-name collision. Check metric_type.

Project structure

├── README.md                        # what this is, and why (you are here)
├── AGENT.md                         # platform behavior verified by testing
├── CODE-EXAMPLES.md                 # every command, Cortex Code and CLI
│
├── dbt_project.yml
├── packages.yml                     # dbt_semantic_view package
├── env.yml                          # dev / staging / prod
├── profiles.yml                     # one profile, reads env_var()
│
├── sql/                             # run these by hand, in order
│   ├── 00_setup.sql                 # demo tables
│   ├── 01_load_demo_data.sql        # LOAD_DEMO_DATA() procedure + call
│   ├── 02_custom_tool.sql           # summarize_issue function
│   └── 03_skill_stage.sql           # stage + PUT + LS verification
│
├── skills/
│   └── triage/SKILL.md              # the workflow the evals measure
│
├── models/
│   ├── sources.yml
│   ├── staging/                     # stg_orders, stg_issues
│   ├── semantic_views/              # sv_workflow (the Analyst tool)
│   └── evaluations/                 # eval_dataset
│
├── agents/
│   └── workflow_agent.sql           # spec-as-macro, with the skills array
│
├── evaluations/
│   ├── AGENT.md                     # how to edit these configs safely
│   ├── eval_answer.yml              # dimension 1, built-in
│   ├── eval_logical_consistency.yml # dimension 2, built-in
│   ├── eval_workflow_routing.yml    # dim 3: built-in TSA + custom routing rules
│   ├── eval_workflow_arguments.yml  # dim 4: built-in TEA + custom arg discipline
│   └── eval_full.yml                # all six metrics, one scorecard
│
├── seeds/
│   └── eval_ground_truth.csv        # 10 rows, 2 expecting no tool call
│
└── macros/
    ├── create_agent.sql / alter_agent.sql
    └── create_eval_stage.sql / run_evaluation.sql

Why the agent spec is inline in a macro

dbt Projects on Snowflake has no runtime file read: both load_file_contents and {% raw %}{% include %}{% endraw %} fail. So each agent's spec lives inside its own deploy_<name> wrapper macro and is passed as text to create_agent / alter_agent, which substitute <<DATABASE>>, <<SCHEMA>>, and <<WAREHOUSE>> for the active target.

Aligning on your workflow

The demo workflow is a placeholder. Retargeting is mechanical once you know the real one. Capture these five things:

1. The workflow steps, in order. What must happen before what. This becomes skills/triage/SKILL.md. The steps are what tool selection scores adherence to.
2. The tools the agent should have, and for each: what it does, what data it touches, when to use it, and when not to. The "when not to" is what makes wrong selection detectable.
3. The mutual exclusions. Which tools must never both fire for one request. Without this, tool selection cannot distinguish thorough from confused.
4. The exact arguments each tool takes, and what a malformed one looks like. This becomes the tool_input field of each ground_truth_invocations entry and drives tool execution accuracy.
5. The requests that should produce NO tool call. Missing information, out of scope, needs a human. Most teams never test these, and they are where workflow agents fail hardest.

Then:

• Replace skills/triage/SKILL.md with the real workflow.
• Rewrite seeds/eval_ground_truth.csv with real questions and a ground_truth_invocations array per row. Keep several rows with "ground_truth_invocations": [].
• Tune the two custom rubrics' "specific failure modes" lists to the real failure modes.
• Swap models/sources.yml and sv_workflow.sql onto real tables.

Scheduling

Run the full config nightly and the scorecard becomes a trend line rather than a snapshot. Two things to remember:

• The task reads whatever config is currently on the stage, so re-upload after every edit.
• Version run_name per run (nightly_20260805), since a repeated name collides with the existing run record.

Task DDL is in CODE-EXAMPLES.md.

References

• Exhibit C: Cortex Agent Evaluations -- illustrated walkthrough of the framework, metrics, traps, and trace behavior this template implements
• Cortex Agent evaluations
• Agent skills
• CREATE AGENT
• CREATE SEMANTIC VIEW
• dbt Projects on Snowflake
• Best practices for building Cortex Agents