Why Evaluate

Cortex Agent evaluations let you test, baseline, and hill-climb on an agent's behavior and performance, so you know when it is ready to roll out. You evaluate against both ground-truth and reference-free metrics, and the agent's activity is traced and monitored so you can see each step on the way to its answer.

The Goal-Plan-Action framework

Instead of judging only the final answer, the metrics follow Snowflake's Goal-Plan-Action (GPA) framework: they evaluate the agent at each stage of its reasoning, so you can pinpoint where it went wrong or was inefficient. The four system metrics trace the loop from the user's goal to the agent's answer.

• Tool selection accuracy covers goal to plan: does the orchestration layer invoke the tools you expect for the user's goal?
• Tool execution accuracy covers plan to actions: does each tool that runs get appropriate input and return output that meets your requirements?
• Answer correctness closes the loop from actions back to goal: how closely does the final response match the expected ground truth?
• Logical consistency spans the whole loop: consistency across instructions, planning, and tool calls. It is reference-free, so it needs no ground truth.

Access to run an evaluation

Evaluations compute their metrics with the AI_COMPLETE function using LLM-as-a-judge, so the role that runs an evaluation needs, among others:

• The SNOWFLAKE.CORTEX_USER database role and the USE AI FUNCTIONS privilege (to call AI_COMPLETE).
• EXECUTE TASK ON ACCOUNT, and USAGE on the databases and schemas holding the agent and the evaluation data.
• USAGE / OWNERSHIP and MONITOR on the agent, and access to every tool the agent uses.
• In Snowsight, USAGE on the warehouse used for the run.

The Metrics

Four system metrics trace the Goal-Plan-Action loop. Three compare against ground truth; logical consistency is reference-free. All are computed by an LLM judge (the AI_COMPLETE function).

System metrics

Custom metrics (LLM-as-a-judge)

Beyond the system metrics, you can define your own. A custom metric supplies a prompt and a scoring range that are passed to the LLM judge along with the run trace. Custom metrics can be ground-truth-based or reference-free, and their prompts can reference trace data with placeholders like {{input}}, {{output}}, {{ground_truth}}, and {{tool_info}}. Use them for domain-specific checks the streamed reply doesn't expose, for example which tables the agent touched.

metrics:
  - "answer_correctness"
  - "logical_consistency"
  - name: "relevance"
    score_ranges:
      min_score: [1, 3]      # low
      median_score: [4, 6]   # medium
      max_score: [7, 10]     # high
    prompt: |
      Rate 1-10 how relevant the response is to the user's query.
      Compare {{output}} against {{ground_truth}} ...

The judge normalizes each score to the range 0.0 to 1.0. Evaluations currently run the judge on claude-4-sonnet using cross-region inference.

The Dataset

An evaluation dataset is a table with two columns: the input query, and a ground-truth VARIANT the judges compare against. What you put in the VARIANT depends on which metrics you enable.

Two keys, two tracks

• ground_truth_output feeds answer_correctness (the "AC track"). It is compared to everything the user sees in the streamed reply.
• ground_truth_invocations feeds tool selection and execution accuracy (the "tool track"). It is an array of expected tool calls, each with tool_name and optional tool_input / tool_output.
• logical_consistency is reference-free and needs no ground truth, so a row can leave the column empty.

One VARIANT can carry keys for several metrics at once, so a single dataset can drive answer correctness, the tool metrics, and custom metrics in the same run. Start with AC-track rows; add tool-track rows once the agent is stable.

Insert a row with PARSE_JSON

CREATE OR REPLACE TABLE agent_evaluation_data (
  input_query VARCHAR,
  ground_truth VARIANT
);

INSERT INTO agent_evaluation_data
  SELECT
    'What was Q1 2025 revenue by product category, and what does our return policy say for electronics?',
    PARSE_JSON('
      {
        "ground_truth_output": "Q1 2025 revenue was ~1.2M Services, 1.4M Hardware, 1.1M Subscriptions USD. Electronics: 30-day returns, 1-year warranty.",
        "ground_truth_invocations": [
          { "tool_name": "finance_analyst",
            "tool_input": "Q1 2025 revenue by product category",
            "tool_output": "SQL aggregating revenue by category for Jan-Mar 2025, ~3 rows near 1.2M/1.4M/1.1M." },
          { "tool_name": "product_docs_search",
            "tool_input": "return policy and warranty for electronics",
            "tool_output": "Policy page mentioning 30-day returns and a 1-year warranty." }
        ]
      }
    ');

Writing good ground truth

Because ground_truth_output is fed to an LLM prompt, treat it as a plain-language rubric built from literal, verifiable values:

• Known, stable answer: state the value with rounding, tolerance, units, and scope ("within ±2% of 123.45; exclude test accounts").
• Live or changing answer: describe what a correct reply should and should not contain, in enough detail that two readers would agree.
• Out-of-scope: state that the agent should refuse and not fabricate.

Run & Inspect

Drive evaluations from SQL, from a YAML config, or from the Snowsight Evaluations tab. Under the hood it is one function to run and a few functions to read results back.

Run with EXECUTE_AI_EVALUATION

One function handles the lifecycle via a verb: 'START', 'STATUS', 'CANCEL', or 'DELETE'. It takes the run name and a stage path to the config YAML.

-- start a run named run-1 from a config on a stage
CALL EXECUTE_AI_EVALUATION(
  'START',
  OBJECT_CONSTRUCT('run_name', 'run-1'),
  '@eval_db.eval_schema.metrics/agent_evaluation_config.yaml'
);

-- check progress. STATUS / CANCEL / DELETE need the object, and take no stage path.
CALL EXECUTE_AI_EVALUATION('STATUS', OBJECT_CONSTRUCT(
  'run_name',    'run-1',
  'object_name', 'eval_db.eval_schema.evaluated_agent',
  'object_type', 'CORTEX AGENT'));

The config YAML

The config has three top-level keys: an optional dataset (to build a dataset from a table), evaluation (which agent + which dataset), and metrics (built-in strings plus any custom definitions).

evaluation:
  agent_params:
    agent_name: "eval_db.eval_schema.evaluated_agent"
    agent_type: "CORTEX AGENT"
  source_metadata:
    type: "dataset"
    dataset_name: "EVALUATION_INPUT"
metrics:
  - "answer_correctness"
  - "logical_consistency"

Read results back

SELECT * FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_EVALUATION_DATA(
  'eval_db', 'eval_schema', 'evaluated_agent', 'CORTEX AGENT', 'run-1'));

Or use the Snowsight Evaluations tab

On an agent's Evaluations tab you get metric trend cards (current average, change vs previous run, and a trend chart), a runs listing, and the ability to compare up to three runs side by side. Opening a record shows three panes: Evaluation results, Thread details, and Trace details.

Iterate & Operate

An evaluation is only useful if it drives the next improvement. The loop: inspect the trace, diagnose the root cause, fix the right layer, and re-run.

Interpret and iterate

Snowsight buckets each metric's records into high (80% or more), medium (30% or more), and failed. Read the pattern, then fix the right layer:

Seed and grow the dataset

• From production: import queries from agent monitoring data, and turn thumbs-up responses into new ground-truth rows.
• With Cortex Code: the cortex-agent skill's sub-skills help you generate synthetic queries (dataset-curation), run a check (evaluate-cortex-agent), diagnose issues (investigate-cortex-agent-evals), and suggest fixes (optimize-cortex-agent).
• Aim for 15 to 20 questions spanning easy, medium, and hard, with phrasing variations of key questions.

Cost

An evaluation runs the agent once per query and then runs LLM judges (the AI_COMPLETE function) to score each metric. You are charged for the agent runs, the judge inference, the warehouse time for the managing tasks and metric queries, and storage for datasets and results.

Known limits to plan around

• Ground-truth staleness: scope input queries to absolute dates ("revenue between January and March 2026"), not relative ones ("this quarter"), so results stay comparable over time.
• Throughput: long traces and many tool calls slow a run. If you hit timeouts, split the dataset (for example by common tool invocation) or shorten a custom-metric prompt.

Traps That Do Not Error

Five configurations the platform accepts without complaint. Each one returns a completed run and a plausible scorecard, so the only way to know your configuration landed as intended is to check. Each trap below comes with its check.

1. The four built-in metric names are reserved

answer_correctness, logical_consistency, tool_selection_accuracy, and tool_execution_accuracy are built-in. If you define a custom metric using one of those names, the built-in takes precedence and your prompt is not used. No error is raised, and the resulting scores look entirely reasonable.

The check, on any eval span:

-- 'custom' = your prompt ran.  'system' = a built-in ran in its place.
RECORD_ATTRIBUTES:"ai.observability.eval.metric_type"::STRING

2. agent_name must be fully qualified

A bare object name is rejected at start, with an error that names the required shape:

Object name WORKFLOW_AGENT format should be 'database.schema.object'

Use MY_DB.MY_SCHEMA.WORKFLOW_AGENT in every config, including single-dimension ones.

3. STATUS needs the object, not just the run

START takes the run name and a stage path. STATUS, CANCEL, and DELETE instead need object_name and object_type alongside run_name, and take no stage path.

CALL EXECUTE_AI_EVALUATION('STATUS', OBJECT_CONSTRUCT(
  'run_name',    'run-1',
  'object_name', 'MY_DB.MY_SCHEMA.WORKFLOW_AGENT',  -- fully qualified
  'object_type', 'CORTEX AGENT'));

All of them also need a session context. Without USE DATABASE, USE SCHEMA, and USE WAREHOUSE you get The DB is not set for the current session.

4. One dataset per source table

Section 4 notes that a dataset: block left in the YAML fails on re-run. The reason is worth knowing, because it bounds the workaround: the dataset version name is a fixed constant.

Dataset version SYSTEM_AI_OBS_CORTEX_AGENT_DATASET_VERSION_DO_NOT_DELETE already exists

Because that name never varies, a second dataset built on the same table collides even under a different dataset_name. In practice there is one dataset per source table. To re-snapshot after changing the rows, either delete the existing dataset or point at a different table.

5. Match the status string exactly

The progression observed across runs:

INVOCATION_IN_PROGRESS → COMPUTATION_IN_PROGRESS → PARTIALLY_COMPLETED → COMPLETED

A run can also park at INVOCATION_PARTIALLY_COMPLETED with rows planned and no errored spans, and not advance. CANCEL and re-run, but check for errored spans first rather than assuming the agent failed.

Reading the Trace

Section 4 reads results through the GET_AI_* functions. This section is the layer underneath: which spans a Cortex Agent turn actually emits, why one tool can appear as two spans or none at all, and how to pull scores directly for cross-run work.

Span taxonomy

Observed in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS across a ten-row run.

Analyst is two spans, not one

SemanticContextTool_Analyst only generates SQL. It never runs it. Execution is always a separate SystemExecuteSQLTool_system_execute_sql plus SqlExecution_SystemSQL pair.

Two consequences for ground truth: one ground_truth_invocations entry for an Analyst tool maps onto a span pair, and the presence of system_execute_sql does not by itself tell you which path ran, because it appears on both.

The verified-query fast path

When a question matches an AI_VERIFIED_QUERIES entry on the semantic view, Analyst does not load the semantic model and emits no Analyst span. It runs the verified SQL directly, and the trace shows only the SQL execution, carrying verified_query_used: true.

Span presence measured across all ten rows of one run:

The three rows with no Analyst span are exactly the three verified queries defined on the semantic view. A one-to-one correlation, so the verified query is the cause.

Pulling scores directly

GET_AI_EVALUATION_DATA in Section 4 is the right tool for one run. For trend work across many runs, query the events table. One structural detail matters: the score and its metadata sit on different records that share an eval_root_id, so each side has to be collapsed before joining or the explanation comes back NULL.

WITH meta AS (
  SELECT RECORD_ATTRIBUTES:"ai.observability.eval.eval_root_id"::STRING AS rid,
         MAX(RECORD_ATTRIBUTES:"ai.observability.eval.metric_name"::STRING) AS metric,
         MAX(RECORD_ATTRIBUTES:"ai.observability.eval.metric_type"::STRING) AS metric_type,
         MAX(RECORD_ATTRIBUTES:"ai.observability.eval.explanation"::STRING)  AS explanation
  FROM SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS
  WHERE RECORD_ATTRIBUTES:"snow.ai.observability.run.name"::STRING = 'run-1'
    AND RECORD_ATTRIBUTES:"ai.observability.eval.metric_name" IS NOT NULL
  GROUP BY 1),
scores AS (
  SELECT RECORD_ATTRIBUTES:"ai.observability.eval.eval_root_id"::STRING AS rid,
         MAX(RECORD_ATTRIBUTES:"ai.observability.eval_root.score"::FLOAT) AS score
  FROM SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS
  WHERE RECORD_ATTRIBUTES:"snow.ai.observability.run.name"::STRING = 'run-1'
    AND RECORD_ATTRIBUTES:"ai.observability.eval_root.score" IS NOT NULL
  GROUP BY 1)
SELECT m.metric, m.metric_type, ROUND(AVG(s.score), 3) AS avg_score,
       MIN(s.score) AS worst
FROM meta m JOIN scores s USING (rid)
GROUP BY 1, 2 ORDER BY 2 DESC, 1;

Judges are not deterministic