> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Scenario tests and evaluations

> What Langy produces when it tests your agent: Scenario multi-turn simulation tests and evaluations, written and run through the langwatch CLI.

Langy writes and runs Scenario simulation tests and evaluations through the `langwatch` CLI, so they live in your project like any test or evaluator you wrote by hand. Which one it picks depends on what the question needs to prove; see the table below.

> [!NOTE]
>
> **Also check:** [Ask Langy](https://langwatch.ai/docs/langy/chat.md) (how you request a test), [Pull
> requests](https://langwatch.ai/docs/langy/pull-requests.md) (the tests it wrote run in your CI), [Agent
> Testing](https://langwatch.ai/docs/agent-testing/overview.md) and [Evaluations](https://langwatch.ai/docs/evaluations/overview.md) (the product surfaces
> these land in).

## Scenario simulation tests

A Scenario test is a multi-turn simulation. A simulated user drives your agent across a full conversation, probing it the way a real user would, and a judge scores the transcript against the criteria you set.

This is how Langy catches behavior that a single-turn check misses: an agent that holds the line for one message but drifts by turn 15.

Langy writes and runs these with:

```bash theme={null}
langwatch scenario create
langwatch scenario run
```

The runs land in Agent Testing > Results, where you can filter, group, and open any run.

![Results overview filtered to red-team runs](https://mintcdn.com/langwatch/6W_w4oAKDAm7lUJ_/images/langy/testing/redteam-overview.png?fit=max\&auto=format\&n=6W_w4oAKDAm7lUJ_\&q=85\&s=14f3e7f97834fc032af4007370067da3)

The results overview filtered to red-team runs. Each row is a scenario run with its outcome.

Open a run to see the judge's verdict and its reasoning against your criteria.

![Scenario verdict and judge reasoning](https://mintcdn.com/langwatch/6W_w4oAKDAm7lUJ_/images/langy/testing/scenario-verdict.png?fit=max\&auto=format\&n=6W_w4oAKDAm7lUJ_\&q=85\&s=aef1ba6221ba40b7be2864aaaac268ef)

A scenario run detail: the judge verdict with the reasoning it used to reach it.

When a run fails, the detail includes the bug log so you can see exactly where the agent broke.

![Failed run detail with bug log](https://mintcdn.com/langwatch/6W_w4oAKDAm7lUJ_/images/langy/testing/failed-run-buglog.png?fit=max\&auto=format\&n=6W_w4oAKDAm7lUJ_\&q=85\&s=a3dfe1b716cf324670dd3ccfe064286a)

A failed-run detail with the bug log showing where the agent's behavior diverged from the criteria.

Read more in the Agent Testing docs: [Overview](https://langwatch.ai/docs/agent-testing/overview.md), [Write scenarios in code](https://langwatch.ai/docs/agent-testing/scenarios-in-code.md), [Red teaming](https://langwatch.ai/docs/agent-testing/red-teaming.md), and [Voice agents](https://langwatch.ai/docs/agent-testing/voice-agents.md).

## Evaluations

An evaluation scores a run against a criterion: answer correctness, a custom rule, an LLM judge, a pairwise comparison. Langy writes evaluators and runs experiments and monitors with the CLI:

```bash theme={null}
langwatch evaluator create
langwatch experiment run
langwatch monitor create
```

Use each for a different job:

* `langwatch experiment run` scores an offline or batch run, for comparing two models or two prompts. See [Experiments](https://langwatch.ai/docs/evaluations/experiments/overview.md) and [running evals in CI/CD](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md).
* `langwatch monitor create` sets up an online evaluation that scores live production traffic. See [Online evaluation](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md).
* `langwatch evaluator create` defines the evaluator both of the above use. See [Evaluators](https://langwatch.ai/docs/evaluations/evaluators/overview.md).

## Which one Langy picks

Langy picks based on what your question needs to prove:

| You want to check | Langy writes |
| - | - |
| The agent handles a full conversation correctly | A Scenario simulation test |
| One run scores well on a criterion | An evaluation |
| Live traffic stays within a criterion | An online monitor |
| Two models or prompts, head to head | An experiment |

You can also name the one you want in the [chat](https://langwatch.ai/docs/langy/chat.md): "write a Scenario test that..." or "set up an evaluator for...".


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.