> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Experiments Overview

> Run a dataset through your prompts and agents in the workbench, score every row with evaluators, and compare runs before you ship a change.

<video controls autoPlay muted loop playsInline width="100%">
  <source src="https://mintcdn.com/langwatch/cmC6VgoJ88MNbY65/media/videos/experiments-workbench.webm?fit=max&auto=format&n=cmC6VgoJ88MNbY65&q=85&s=c972c94a643e2a41328b386d1dcc3c3c" type="video/webm" />
</video>

The clip above builds an experiment on a customer support dataset: a prompt column runs the ten rows, an LLM Answer Match evaluator scores each answer against the expected output, and the column header reads the pass rate, 70 percent on this run.

> [!TIP]
>
> **Let your agent set this up.** [Copy the evaluations prompt](https://langwatch.ai/docs/skills/directory.md) into your coding agent to get started automatically.

An experiment is a table. Each row is a dataset entry, and each column is a prompt, an agent, a comparison or an evaluator that runs on that entry. An evaluator attached to a prompt or an agent column scores the output of that column. You build the table in the workbench, run it, and read the pass rate per column.

![The experiments workbench with a dataset, a prompt column with outputs and an evaluator chip with pass and fail results](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/workbench-overview.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=2550ada321c85a852f50d565feabe863)

## When to use experiments

* Check that a prompt change does not regress quality before you deploy it.
* Compare two prompts, two models or two agent configurations on the same dataset.
* Gate a deployment on a quality threshold from your CI pipeline.
* Test an evaluator itself against a dataset with known answers before you use it in production.

## Create an experiment

1. Open [Experiments](https://app.langwatch.ai/@project/experiments) and click **New Experiment**, then **Create Experiment**. The workbench opens with a generated name. Click the name to rename it.
2. Under **Datasets**, click **Add** and pick **Select existing dataset**, **Upload CSV** or **Create new**. You can edit cells inline. A dataset created inside the workbench stays there until you pick **Save as dataset** from its tab menu.
3. Under **Prompts or Agents**, click **Add**. The **Add to Evaluation** drawer offers four columns: **Prompt**, **Agent**, **Comparison** and **Evaluator**.
4. On a prompt or agent column, click **Add evaluator** and pick a saved evaluator, or create one from the evaluator categories: **Expected Answer**, **LLM as Judge**, **RAG Quality**, **Quality Aspects**, **Safety**, **Custom (Code)** and **Custom (from Workflow)**.

![The Choose Evaluator Category drawer with the five categories and the two custom entries](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/workbench-evaluator-category.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=5e45d3be6577117353f9de163cada50e)

![The Add to Evaluation drawer with the Prompt, Agent, Comparison and Evaluator cards](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/workbench-add-to-evaluation.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=05fae5c60588c6b59fa3931cb9ac6d5a)

| Column | What it runs |
| - | - |
| Prompt | A versioned prompt from your prompt library with the model you pick in the editor. Prompt variables such as `{{input}}` map to dataset columns in the **Variables** section. |
| Agent | An agent you connected to LangWatch, an HTTP endpoint, a code agent or a workflow. |
| Comparison | An LLM judge that reads the outputs of two or more columns for the same row and picks the best one. See [Comparison](https://langwatch.ai/docs/evaluations/experiments/ui/pairwise-compare.md). |
| Evaluator | An evaluator run directly on the dataset, to test the evaluator against rows with known answers. |

The workbench autosaves. **History** lists the saved versions, and **Restore** writes an old setup forward as a new version.

## Map image and file columns

A dataset column of type `image` or `file` maps to a prompt input of type **Image** or **File**, and to an agent input. A connected agent column also has an optional **attachment** input for the same columns.

A prompt gets the file as an image, an audio or a document content part. A plain text or JSON file is different: LangWatch reads it and sends its text inline, under the file name, rather than as a file content part. An agent gets the file as a base64 data URL, so it never has to open a LangWatch address. See [Images and files in datasets](https://langwatch.ai/docs/datasets/dataset-images.md).

## Map evaluator inputs

Every evaluator declares the fields it reads, for example `input`, `output`, `expected_output` or `contexts`. The **Variables** section of the evaluator editor maps each field to a source: the output of the column it is attached to, or a dataset column.

LangWatch fills the mapping for you when the names match. A dataset column named `expected_output`, `expected_answer` or `ground_truth` maps to the evaluator's `expected_output` field; the column output maps to `output`. A field that stays unmapped shows **Required**, and the run button reads **Configure missing mappings first** until you map it.

## Run

**Run** in the header runs every column on every row. The play button on a column header runs that column only, the button on a cell runs one row, and the toolbar that appears when you select rows runs the selection. While a run is in progress the button reads **Stop**.

**Run Options** sets the concurrency and the row height, and **Automation** shows the request that triggers this experiment from a pipeline. See [Run experiments from CI/CD](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md).

## Read the results

Each cell shows the column output, and each evaluator chip shows its verdict. Click a chip to open its **Result** block with the status, the score or the label when the evaluator returns one, and the details, for example the reason an LLM judge gave. The same popover has **Rerun**, **Run on all rows** and **Edit Configuration**.

The column header shows the pass rate for the column. Hover over it to open the summary: **Rows**, **Pass Rate**, **Avg Latency**, **Total Cost**, **Execution Time**, and one line per evaluator with its pass rate or average score.

![The column summary popover with rows, pass rate, average score, latency, cost and the per-evaluator breakdown](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/workbench-column-summary.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=f8b7132d89046a5395287c4d840ebaf5)

## Compare runs

**Results** in the header opens the results page for the experiment. The **Experiment Runs** sidebar lists every run, newest first. Tick two runs to see them side by side: one chart per metric (cost, latency, and the pass rate of each evaluator) and, per row, both outputs with their evaluator chips, so you can check that a change fixed the failing rows without breaking the passing ones.

![The results page with two experiment runs compared side by side](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/experiment-runs-compare.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=4176424542860cd1a2a7fa4b82844f27)

## Run experiments from code

The Python and TypeScript SDKs run the same loop from a notebook or a script: load a dataset, call your application on each row, and log evaluator results to an experiment in LangWatch. See [Experiments via SDK](https://langwatch.ai/docs/evaluations/experiments/sdk.md).

## Next steps

> [card] [Answer correctness](https://langwatch.ai/docs/evaluations/experiments/ui/answer-correctness.md)
>
> Score outputs against expected answers in your dataset.

> [card] [LLM-as-a-judge](https://langwatch.ai/docs/evaluations/experiments/ui/llm-as-a-judge.md)
>
> Score outputs against written criteria when there is no expected answer.

> [card] [Comparison](https://langwatch.ai/docs/evaluations/experiments/ui/pairwise-compare.md)
>
> Judge two or more prompts, models or agents head-to-head.

> [card] [Run from CI/CD](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md)
>
> Trigger an experiment from your pipeline and read the results back.

> [card] [Prompt optimization with Langy](https://langwatch.ai/docs/langy/prompt-optimization.md)
>
> Langy scores, rewrites and reruns your prompt in one loop.

> [card] [Datasets](https://langwatch.ai/docs/datasets/overview.md)
>
> Create and manage the datasets your experiments run on.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.