> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.
# Evaluators
> An evaluator attached to a test suite runs after every scenario run and reports beside the judge's verdict, such as a faithfulness check on a RAG agent, an answer clarity score, a PII scan, or a comparison against a golden answer the scenario carries. A required evaluator fails the run when it fails.
## When to attach an evaluator
The judge grades behavior from the criteria. An evaluator adds a check the [evaluator library](/docs/evaluations/evaluators/overview) already does well: whether the answer of a RAG agent is faithful to the contexts it retrieved, a score for the clarity of the answer, a PII scan of the transcript, or a comparison against a golden answer the scenario carries. Any evaluator of the project can be attached, LLM judges included.
For a golden answer, the test suite declares [fields](/docs/agent-testing/test-suites#fields), for example `reference_answer`, and each scenario carries its value. A blank value skips the evaluators that read it.
## Attach an evaluator
**Edit suite** opens the suite editor. Under **Customize test suite**, **Add evaluator** lists the evaluators of the project, and picking one attaches it with its mappings inferred. When a required input is still unmapped, or the evaluator reads an expected value, the drawer opens on the attachment so you confirm each mapping.
A chip with an amber alert has a required input with no mapping. **Run suite** does not start in that state: it opens the drawer on that evaluator instead.
## Mappings
Every evaluator input reads from one source:
| Source | Paths | What it reads |
| - | - | - |
| **Conversation** | `first_user_message`, `last_agent_message`, `transcript`, `messages` | The first message of the simulated user, the last reply of the agent, the whole conversation as `role: content` lines, or the messages as JSON. |
| **Scenario** | `situation`, `criteria`, `fields.` | The scenario's own text, or its value for a field the suite declares. |
| **Trace** | `contexts`, `spans`, `tool_calls..input`, `tool_calls..output` | The retrieved contexts of the run's traces, every span of those traces as JSON in start order, or the input or output of the last call of a tool. Needs [linked traces](/docs/agent-testing/linking-your-traces). |
| A value | | A literal, the same for every scenario. |
Mappings are inferred from the input names: `input`, `question` or `user_input` reads the first user message; `output`, `response` or `answer` reads the last agent message; `transcript`, `conversation` or `messages` reads the transcript; `contexts` or `retrieved_contexts` reads the trace's contexts. So a faithfulness evaluator on a RAG agent needs no mapping at all once its traces are linked.
An `expected_*` input reads a suite field: the field with the same name, otherwise the one field whose name shares a word with the input. `expected_output` also takes `golden`, `answer`, `sql`, `query`, `label` or `target`, and `expected_contexts` takes `schema`, `context` or `table`. A suite with a single field maps it regardless, and two matches leave the input for you to pick in the drawer. A tool call or the spans are never inferred: set them by hand.
For example, the drawer above reads `output` from the `run_sql` tool call and the two expected inputs from the `golden_sql` and `table_schema` fields, one mapping per input under **Variables**. `spans` hands an evaluator the whole run, every span of every trace as JSON, when one tool call at a time is not enough.
## Required evaluators and scores
**Required to pass** makes a failing result fail the scenario. It is on by default for an evaluator with a pass or fail verdict. A score-only evaluator, such as an answer clarity score, reports beside the verdict and never gates.
## What each run reports
After the judge, every attached evaluator runs and the run keeps one result per evaluator:
| Status | Meaning |
| - | - |
| `passed`, `failed` | The evaluator's verdict. A failed required evaluator fails the scenario, and the verdict line names it. A required input the trace never delivered, no such tool call, no retrieved contexts or no spans, is recorded as `failed` with the reason naming it. |
| `scored` | A score, for an evaluator with no pass or fail verdict. |
| `skipped` | A field the mapping reads is blank on this scenario. The reason names the field. |
| `error` | The evaluator itself failed to run. The reason says why. |
Between the judge's verdict and the evaluator results the run reads `Evaluating`, and `langwatch test-suite run --wait` treats it as still going. A required evaluator that never ran fails the run instead of leaving it pending. The run header shows one pill per evaluator with its pass rate or its mean score, and the run drawer lists the rows under the criteria. See [Results](/docs/agent-testing/results#evaluator-results).
## Evaluators on a run plan
The [run dialog](/docs/agent-testing/run-plans#the-run-dialog) lists the evaluators inherited from the test suites in scope, and **Add evaluator** under **On this run** adds one to the plan itself. A plan evaluator reads the conversation and the trace only, because a plan may cover suites with different fields. Use it for a check that applies to every scenario, such as a PII scanner.
## Step by step: a golden answer on every scenario
The screenshots follow a text-to-SQL suite, where the golden answer is a query and the evaluator is **SQL Query Equivalence**. The steps are the same for any evaluator that compares against a field, such as a reference answer on a RAG agent.
On the **Scenarios** tab, press **Edit suite** and, under **Customize test suite**, **Add fields**. Add one field per value the evaluator compares against, here `golden_sql` and `table_schema` of type **Text**, then **Save**. The editor of every scenario in the suite now asks for them.
Open a scenario with **Edit** in its row menu. After the criteria, the editor shows one box per field. Fill them and **Save**. A blank value is allowed: the evaluators that read it are skipped for that scenario.
Back in the suite editor, press **Add evaluator** and pick the evaluator. The drawer opens with the expected inputs inferred from the fields. Here `output` has to read the query the agent ran rather than its reply, so change it to the **Trace** source, tool call `run_sql`, `input`. Leave **Required to pass** on and press **Save Changes**, then **Save** on the suite.
A score-only evaluator, such as an answer clarity judge, is attached the same way. Its switch is disabled with `Scores report, they do not gate.`
Press **Run suite**, pick the agent and press **Run N scenarios**. The dialog lists the inherited evaluators, and **Add evaluator** under **On this run** adds a check for this run only. While the evaluators run, a row reads `Evaluating` in the **Result** column and `Grading` in the **Evaluators** column.
The run header shows one pill per evaluator, and every row shows one pill per evaluator with its verdict, its score or `Skipped`. Press a row: the drawer lists the evaluators under **EVALUATORS**, below the criteria, with the `REQUIRED` mark, the explanation and the **Inputs** it compared. A run the required evaluator failed reads `Verdict: FAILED ยท SQL Query Equivalence`, with the judge's criteria kept.
## Troubleshooting
### The evaluator was skipped
The scenario has no value for the field the mapping reads. The reason names it, for example `no golden_sql on this scenario`, and the run keeps the judge's verdict. Fill the field, then **Rerun this scenario** from the row menu.
### Run suite opens the evaluator drawer instead of running
A required input has no mapping. The chip shows an amber alert, and **Run** opens the drawer on that evaluator with `Please map all required fields: expected_output`. Map the input, press **Save Changes** and run again. The API refuses the same run with `suite_evaluator_mappings_missing`.
### The evaluator failed with a reason about the trace
`no run_sql call in the trace`, `no retrieved contexts in the trace` or `no spans in the trace` means the run's traces did not hold what the mapping reads. The platform waits for the traces and retries, and on the last attempt records the evaluator as `failed`, so a required one fails the run.
Check that the traces reach the project and join the run, see [Linking your traces](/docs/agent-testing/linking-your-traces), and that the tool name in the mapping is the span name the agent reports.
## From the CLI and the API
```bash theme={null}
langwatch test-suite create "Case lookups" \
--field golden_sql:text --field table_schema:text \
--evaluator sql-query-equivalence --required
langwatch scenario create "Open cases count" \
--situation "The user is a support lead who wants to know how many support cases are open." \
--criteria "Gives a number of open cases" \
--test-suite "Case lookups" \
--field golden_sql="SELECT COUNT(*) FROM cases WHERE status = 'open'" \
--field table_schema="CREATE TABLE cases (id INT, status TEXT, created_at TIMESTAMP)"
langwatch test-suite update "Case lookups" --evaluators-json evaluators.json
langwatch run-plan run --test-suite "Case lookups" --target http:agent_abc123 --evaluator pii-leak-scanner
langwatch simulation-run get --format json # results.evaluations
```
`--evaluator` infers the mappings the way the suite editor does. `--evaluators-json` takes the full attachment list, which is how a tool call mapping is set:
```json theme={null}
[{
"id": "attachment_abc123",
"evaluatorId": "evaluator_abc123",
"required": true,
"mappings": {
"output": { "type": "source", "sourceId": "trace", "path": ["tool_calls", "run_sql", "input"] },
"expected_output": { "type": "source", "sourceId": "scenario", "path": ["fields", "golden_sql"] },
"expected_contexts": { "type": "source", "sourceId": "scenario", "path": ["fields", "table_schema"] }
}
}]
```
The same list travels as `evaluators` on `POST /api/v1/test-suites`, `PATCH /api/v1/test-suites/{id}` and `config.evaluators` on `POST /api/v1/run-plans/run`, and `GET /api/simulation-runs/{scenarioRunId}` answers with `results.evaluations`. See the [Test Suites API reference](/docs/api-reference/test-suites/overview) and the [Run Plans API reference](/docs/api-reference/run-plans/overview). The Python SDK takes `fields` and `evaluators` on `test_suites.create`, `test_suites.update` and `run_plans.run`; the TypeScript SDK takes them on `testSuites.create`, `testSuites.update` and, under `config`, on `runPlans.run`. The MCP tools `platform_create_test_suite`, `platform_update_test_suite` and `platform_run_plan` take the same.
## From code
A scenario written with the Scenario SDK attaches evaluators in the test itself, with the same mappings and the same gate, and its results appear on the platform beside the judge's verdict. See [Evaluators on Scenarios](https://langwatch.ai/scenario/advanced/evaluators) in the Scenario documentation.
Also check: [Test suites](/docs/agent-testing/test-suites), [Scenarios](/docs/agent-testing/scenarios), [Results](/docs/agent-testing/results), [Linking your traces](/docs/agent-testing/linking-your-traces).
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.