> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Test Suites

> A test suite is a named group of scenarios you run together. It defines the fields a scenario may carry a value for and the evaluators that run after each scenario. The targets, the models and the repeat count are set on the run plan.

## The rail

The **Scenarios** tab lists the test suites of the project under **Test Suites** in the rail on the left. Press one to open its scenarios in the table. **New Test Suite** asks for a name only.

![The test suite rail with the row menu of the Refunds test suite open](https://mintcdn.com/langwatch/vSvONwxpUQtFi1t0/images/agent-testing/test-suite-rail-menu.png?fit=max\&auto=format\&n=vSvONwxpUQtFi1t0\&q=85\&s=f5a32c88e5311df37757f3507820d0c9)

The row menu of a test suite offers:

| Action | What it does |
| - | - |
| **New scenario** | Opens the editor with this test suite preselected. |
| **Run suite** | Opens the run dialog with every scenario of the test suite in scope. |
| **Edit** | Opens the suite editor: the name, the fields and the evaluators. Past runs keep the old name. |
| **Open recent runs** | Lists the latest runs of this test suite. |
| **Archive suite** | Archives the test suite and the scenarios in it. Test runs are preserved. |

Every project has a **Default** test suite. A scenario saved with **No test suite** goes there.

Under the suite name, the header shows a chip per field and a chip per evaluator, and the count line `N scenarios · N fields · N evaluators`.

## Fields

A test suite reads like a dataset. Beyond the situation and the criteria, it declares typed fields, and a scenario in it may carry a value for each. A blank value skips the evaluators that read it. A field holds what an [evaluator](https://langwatch.ai/docs/agent-testing/evaluators.md) compares against: the golden SQL query, the reference answer, the tables the answer had to be written against.

**Edit suite** opens the suite editor. Under **Customize test suite**, **Add fields** opens the field rows: an identifier and a type each, with **Add field** for the next one. Drag a row by the handle on its left to reorder the fields. The handle appears once the suite declares more than one field, and it takes the arrow keys when it has focus.

![The Edit test suite drawer with the field rows golden\_sql and table\_schema, each with a drag handle, a type of Text and a remove button, and the evaluator chips under them](https://mintcdn.com/langwatch/OHUpQQmmdnh4pjQ7/images/agent-testing/suite-editor-evaluators.png?fit=max\&auto=format\&n=OHUpQQmmdnh4pjQ7\&q=85\&s=a0ce64c2ad3f2f4e9e51a8ee2b6c4936)

The field rows of the `Case lookups` suite, with the drag handle on the left of each.

| Setting | Rule |
| - | - |
| **Identifier** | Lowercase letters, digits and underscores, starting with a letter, up to 64 characters, for example `golden_sql`. `situation`, `criteria`, `name`, `input` and `output` are reserved. |
| **Type** | `text`, `number` or `boolean`. A text value is a growing text box on the scenario; a number a number input; a boolean a switch. |

A suite declares up to 30 fields. A field an attached evaluator still reads cannot be removed: detach the evaluator or change its mapping first. Removing a field keeps the values scenarios hold for it; the scenario editor lists them under **Not in this suite** until they are removed.

## Evaluators

The evaluators attached to a suite run after every scenario run and report beside the judge's verdict. A required pass or fail evaluator fails the scenario when it fails, and a score-only evaluator never gates. See [Evaluators](https://langwatch.ai/docs/agent-testing/evaluators.md) for the mappings, the gate and what each run reports.

## Run a test suite

**Run suite**, in the rail menu or at the top of the table, opens the [run dialog](https://langwatch.ai/docs/agent-testing/run-plans.md#the-run-dialog). The run goes under the run plan named after the test suite and the agent, for example `Refunds checkout-prod`, so every run of the same test suite against the same agent joins one history on the Results tab.

## Move scenarios between test suites

**Move to suite...** in the row menu of a scenario opens **Move to test suite**, which lists the test suites of the project. Select several rows and press **Move to suite** in the bar that appears to move them together. A scenario belongs to exactly one test suite.

![The Move to test suite dialog with the test suites of the project](https://mintcdn.com/langwatch/vSvONwxpUQtFi1t0/images/agent-testing/move-to-test-suite.png?fit=max\&auto=format\&n=vSvONwxpUQtFi1t0\&q=85\&s=30ad1dbfc50a88f57f3ec7ab22daa117)

## Test suites and labels

A test suite is the group you run before a release. A label is a tag on a scenario, and a scenario can have any number of them. A run plan can scope by labels across every test suite, for example every scenario labelled `smoke`, so use labels for a cut that crosses the test suites and a test suite for the group that ships together.

## From Code

Scenarios written with the Scenario SDK and run from your own test runner report to the project and appear under **From Code** in the rail, one entry per `setId`. Their table is read-only and shows each scenario with its last run. Edit them in your repository. The platform lists them so their results appear next to the test suites written here.

![A From Code test suite, with its scenarios and their last runs](https://mintcdn.com/langwatch/vSvONwxpUQtFi1t0/images/agent-testing/from-code-suite.png?fit=max\&auto=format\&n=vSvONwxpUQtFi1t0\&q=85\&s=5fa5301985ffb91f873136209175b49c)

## From the CLI and the API

```bash theme={null}
langwatch test-suite create "Case lookups" --field golden_sql:text --field table_schema:text
langwatch test-suite update "Case lookups" --field golden_sql:text --field table_schema:text --field row_limit:number
langwatch test-suite get "Case lookups" --format json
```

`--field identifier:type` repeats, and on `update` the flags together replace the list the suite declares. The same list travels as `fields` on `POST /api/v1/test-suites` and `PATCH /api/v1/test-suites/{id}`; see the [Test Suites API reference](https://langwatch.ai/docs/api-reference/test-suites/overview.md). The Python and TypeScript SDKs take `fields` on `test_suites.create` / `testSuites.create` and `test_suites.update` / `testSuites.update`; the MCP tools `platform_create_test_suite` and `platform_update_test_suite` take the same.

Also check: [Write scenarios in code](https://langwatch.ai/docs/agent-testing/scenarios-in-code.md), [Scenarios](https://langwatch.ai/docs/agent-testing/scenarios.md), [Evaluators](https://langwatch.ai/docs/agent-testing/evaluators.md), [Run plans](https://langwatch.ai/docs/agent-testing/run-plans.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.