> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Experiments via SDK

> Run experiments programmatically from notebooks or scripts to batch test your LLM applications.

> [!TIP]
>
> **Let your agent set this up.** [Copy the evaluations prompt](https://langwatch.ai/docs/skills/directory.md) into your coding agent to get started automatically.

The LangWatch SDK runs experiments from your Python or TypeScript code and tracks
each run in LangWatch.

Not using Python or TypeScript? The SDKs call the same REST endpoints you can call
yourself: [create an experiment](https://langwatch.ai/docs/api-reference/experiments/create-an-experiment.md),
[run it](https://langwatch.ai/docs/api-reference/experiments/run-an-experiment.md), then
[poll the run](https://langwatch.ai/docs/api-reference/experiments/poll-a-run.md) and
[read its results](https://langwatch.ai/docs/api-reference/experiments/read-run-results.md).

## Quickstart

### 1. Install the SDK

> [tab] Python
>
> ```bash theme={null}
> pip install langwatch
> ```

> [tab] TypeScript
>
> ```bash theme={null}
> npm install langwatch
> # or
> pnpm add langwatch
> ```

### 2. Set your API Key

> [tab] Python (Notebook)
>
> ```python theme={null}
> import langwatch
>
> langwatch.login()
> ```
>
> Be sure to log in or create an account using the displayed link, then provide your API key when prompted.

> [tab] Environment Variable
>
> ```bash theme={null}
> export LANGWATCH_API_KEY=your_api_key
> export LANGWATCH_PROJECT_ID=your_project_id  # Required for service API keys
> ```
>
> > [!NOTE]
> >
> > `LANGWATCH_PROJECT_ID` is required when using a **service API key** (e.g. for CI/CD or multi-project setups). Project API keys obtained via `langwatch.login()` or from the project settings page already have the project context built in.

### 3. Start tracking

> [tab] Python
>
> ```python theme={null}
> import langwatch
> import pandas as pd
>
> # Load your dataset
> df = pd.read_csv("my_dataset.csv")
>
> # Initialize a new experiment
> evaluation = langwatch.experiment.init("my-experiment")
>
> # Wrap your loop with evaluation.loop(), and iterate as usual
> for idx, row in evaluation.loop(df.iterrows()):
>     # Run your model or pipeline
>     response = my_agent(row["question"])
>
>     # Log a metric for this sample
>     evaluation.log("sample_metric", index=idx, score=0.95)
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> import { LangWatch } from 'langwatch';
>
> // Initialize the SDK
> const langwatch = new LangWatch();
>
> // Your dataset
> const dataset = [
>   { question: "What is 2+2?", expected: "4" },
>   { question: "What is the capital of France?", expected: "Paris" },
> ];
>
> // Initialize evaluation
> const evaluation = await langwatch.experiments.init("my-experiment");
>
> // Run evaluation with a callback
> await evaluation.run(dataset, async ({ item, index }) => {
>   // Run your model or pipeline
>   const response = await myAgent(item.question);
>
>   // Log a metric for this sample
>   evaluation.log("sample_metric", { index, score: 0.95 });
> });
> ```

Your evaluation metrics are now tracked and visualized in LangWatch.

![Evaluation Results Sample](https://mintcdn.com/langwatch/iJjBH4X_YNQ578jk/images/offline-evaluation/evaluation-sample.png?fit=max\&auto=format\&n=iJjBH4X_YNQ578jk\&q=85\&s=7a6443ef0a813dcaeb4b2c9dce9089ab)

## Core Concepts

### Evaluation Initialization

The evaluation is started by creating an evaluation session with a descriptive name:

> [tab] Python
>
> ```python theme={null}
> evaluation = langwatch.experiment.init("rag-pipeline-openai-vs-claude")
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> const evaluation = await langwatch.experiments.init("rag-pipeline-openai-vs-claude");
> ```

### Iterating over data

> [tab] Python
>
> Use `evaluation.loop()` around your iterator so the entries are tracked:
>
> ```python theme={null}
> for index, row in evaluation.loop(df.iterrows()):
>     # Your existing evaluation code
> ```

> [tab] TypeScript
>
> Use `evaluation.run()` with a callback that receives each item:
>
> ```typescript theme={null}
> await evaluation.run(dataset, async ({ item, index, span }) => {
>   // Your existing evaluation code
> });
> ```
>
> The callback receives `item` (the current dataset item), `index` (the current index), and `span` (an OpenTelemetry span for custom tracing).

### Metrics logging

Track any metric you want with `evaluation.log()`:

> [tab] Python
>
> ```python theme={null}
> # Numeric scores
> evaluation.log("relevance", index=index, score=0.85)
>
> # Boolean pass/fail
> evaluation.log("contains_citation", index=index, passed=True)
>
> # Include additional data for debugging
> evaluation.log("coherence", index=index, score=0.9,
>                data={"output": result["text"], "tokens": result["token_count"]})
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> // Numeric scores
> evaluation.log("relevance", { index, score: 0.85 });
>
> // Boolean pass/fail
> evaluation.log("contains_citation", { index, passed: true });
>
> // Include additional data for debugging
> evaluation.log("coherence", {
>   index,
>   score: 0.9,
>   data: { output: result.text, tokens: result.tokenCount }
> });
> ```

## Comparing Multiple Targets

When comparing different models, prompts, or configurations, use targets to organize your results.
Both SDKs provide a `target()`, `withTarget()` context that automatically captures latency and enables context inference.
Once a row's targets have each produced an output, [Comparison](#comparison) judges them against each other and names a winner.

> [tab] Python
>
> Use `evaluation.target()` for automatic latency capture and context inference:
>
> ```python theme={null}
> import langwatch
>
> evaluation = langwatch.experiment.init("model-comparison")
>
> for index, row in evaluation.loop(df.iterrows()):
>     def compare_models(index, row):
>         # Evaluate gpt-6-luna with automatic latency tracking
>         with evaluation.target("gpt6-baseline", {"model": "openai/gpt-6-luna"}):
>             response = call_openai("gpt-6-luna", row["question"])
>             evaluation.log_response(response)  # Store the model output
>             # Target is auto-inferred inside target()!
>             evaluation.log("accuracy", index=index,
>                           score=calculate_accuracy(response, row["expected"]))
>
>         # Evaluate Claude with automatic latency tracking
>         with evaluation.target("claude-experiment", {"model": "anthropic/claude-4-opus"}):
>             response = call_anthropic("claude-4-opus", row["question"])
>             evaluation.log_response(response)
>             evaluation.log("accuracy", index=index,
>                           score=calculate_accuracy(response, row["expected"]))
>
>     evaluation.submit(compare_models, index, row)
> ```
>
> > [!NOTE]
> >
> > `evaluation.target()` automatically captures latency, creates isolated traces per target, and
> > enables context inference so `log()` calls don't need explicit `target` parameters. Use
> > `log_response()` to store the model's output.
>
> Alternatively, use the `target` parameter directly with `evaluation.log()`:
>
> ```python theme={null}
> evaluation.log(
>     "accuracy",
>     index=index,
>     score=0.95,
>     target="gpt6-baseline",
>     metadata={"model": "openai/gpt-6-luna", "temperature": 0.7}
> )
> ```

> [tab] TypeScript
>
> Use `withTarget()` for automatic latency capture and context inference:
>
> ```typescript theme={null}
> import { LangWatch } from "langwatch";
>
> const langwatch = new LangWatch();
>
> const evaluation = await langwatch.experiments.init("model-comparison");
>
> await evaluation.run(dataset, async ({ item, index }) => {
>   // Run targets in parallel with automatic tracing
>   const [gpt6Result, claudeResult] = await Promise.all([
>     evaluation.withTarget("gpt6-baseline", { model: "openai/gpt-6-luna" }, async () => {
>       const response = await callOpenAI("gpt-6-luna", item.question);
>       // Target and index are auto-inferred inside withTarget()!
>       evaluation.log("accuracy", { score: calculateAccuracy(response, item.expected) });
>       return response;
>     }),
>
>     evaluation.withTarget("claude-experiment", { model: "anthropic/claude-4-opus" }, async () => {
>       const response = await callAnthropic("claude-4-opus", item.question);
>       evaluation.log("accuracy", { score: calculateAccuracy(response, item.expected) });
>       return response;
>     }),
>   ]);
>
>   // Latency is automatically captured from each withTarget() span
>   console.log(`gpt-6-luna: ${gpt6Result.duration}ms, Claude: ${claudeResult.duration}ms`);
> });
> ```
>
> > [!NOTE]
> >
> > `withTarget()` automatically captures latency, creates isolated traces per target, and enables context inference so `log()` calls don't need explicit `target` or `index` parameters.

### Target Registration

The first time you use a target name, it's automatically registered with the provided metadata:

> [tab] Python
>
> ```python theme={null}
> # Using target() - metadata is set when entering the context
> with evaluation.target("gpt6", {"model": "gpt-6-luna", "temp": 0.7}):
>     evaluation.log_response("AI response here")  # Store the output
>     evaluation.log("latency", index=0, score=150)  # target auto-inferred
>     evaluation.log("accuracy", index=0, score=0.95)  # target auto-inferred
>
> # Or using explicit target parameter (without target() context)
> evaluation.log("latency", index=0, target="gpt6", metadata={"model": "gpt-6-luna", "temp": 0.7})
>
> # Subsequent calls can omit metadata - it's already registered
> evaluation.log("accuracy", index=0, target="gpt6", score=0.95)
> evaluation.log("latency", index=1, target="gpt6", score=150)
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> // Using withTarget() - metadata is set once when registering the target
> await evaluation.withTarget("gpt6", { model: "gpt-6-luna", temp: 0.7 }, async () => {
>   evaluation.log("latency", { score: 150 });  // target auto-inferred
>   evaluation.log("accuracy", { score: 0.95 }); // target auto-inferred
> });
>
> // Or using explicit target parameter
> evaluation.log("latency", { index: 0, target: "gpt6", metadata: { model: "gpt-6-luna", temp: 0.7 } });
> evaluation.log("accuracy", { index: 0, target: "gpt6", score: 0.95 }); // metadata already registered
> ```

> [!WARNING]
>
> If you provide different metadata for the same target name, an error will be raised. Use a
> different target name if you want different configurations.

### Metadata for Comparison

Target metadata is used for comparison charts in the LangWatch UI. You can group results by any metadata field:

> [tab] Python
>
> ```python theme={null}
> # Compare different temperatures
> for temp in [0.0, 0.5, 0.7, 1.0]:
>     for index, row in evaluation.loop(df.iterrows()):
>         response = call_llm(row["question"], temperature=temp, reasoning_effort="none")
>         evaluation.log(
>             "quality",
>             index=index,
>             score=evaluate_quality(response),
>             target=f"temp-{temp}",
>             metadata={"model": "gpt-6-luna", "temperature": temp}
>         )
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> // Compare different temperatures
> for (const temp of [0.0, 0.5, 0.7, 1.0]) {
>   await evaluation.run(dataset, async ({ item, index }) => {
>     const response = await callLLM(item.question, { temperature: temp, reasoning_effort: "none" });
>     evaluation.log("quality", {
>       index,
>       score: evaluateQuality(response),
>       target: `temp-${temp}`,
>       metadata: { model: "gpt-6-luna", temperature: temp }
>     });
>   });
> }
> ```

In the LangWatch UI, you can then visualize how quality varies across temperature values.

## Parallel Execution

LLM calls can be slow. Both SDKs support parallel execution to speed up your evaluations.

> [tab] Python
>
> Use the built-in parallelization by putting the content of the loop in a function and submitting it:
>
> ```python {4,8} theme={null}
> import langwatch
>
> evaluation = langwatch.experiment.init("parallel-eval-example")
>
> for index, row in evaluation.loop(df.iterrows(), threads=4):
>     def task(index, row):
>         result = agent(row["question"])  # Runs in parallel
>         evaluation.log("response_quality", index=index, score=0.92)
>
>     evaluation.submit(task, index, row)
> ```
>
> > [!NOTE]
> >
> > By default, `threads=4`. Adjust based on your API rate limits and system resources.
>
> ### Async-native mode
>
> The default `loop()`, `submit()` path above already parallelises, each submitted task runs in a worker thread, so sync and async tasks both speed up with no extra work on your side. That's the right choice for most users.
>
> Reach for `aloop()`, `asubmit()` only when your code is fully async-first and your task relies on async state whose identity is tied to one event loop. The threading path spins up a fresh event loop per worker, so those objects raise `"Future attached to a different loop"` on first use. `aloop`, `asubmit` keep every submitted task on the caller's event loop, so that state stays valid across concurrent items.
>
> ```python theme={null}
> import langwatch
>
> evaluation = langwatch.experiment.init("async-eval-example")
>
> async def task(index, row):
>     result = await my_async_agent(row["question"])
>     evaluation.log("response_quality", index=index, score=0.92)
>
> index = 0
> async for row in evaluation.aloop(dataset, concurrency=4):
>     evaluation.asubmit(task, index, row)
>     index += 1
> ```
>
> Sync callables passed to `asubmit` are automatically offloaded to a worker thread so they don't block the event loop for concurrent async siblings.

> [tab] TypeScript
>
> Pass the `concurrency` option to control how many items run in parallel:
>
> ```typescript theme={null}
> await evaluation.run(
>   dataset,
>   async ({ item, index }) => {
>     const result = await agent(item.question); // Runs in parallel
>     evaluation.log("response_quality", { index, score: 0.92 });
>   },
>   { concurrency: 4 },
> );
> ```
>
> > [!NOTE]
> >
> > By default, `concurrency=4`. Adjust based on your API rate limits and system resources.

## Built-in Evaluators

LangWatch provides a library of evaluation metrics out of the box.

> [tab] Python
>
> Use `evaluation.evaluate()` with pre-built evaluators:
>
> ```python theme={null}
> for index, row in evaluation.loop(df.iterrows()):
>     def task(index, row):
>         response, contexts = execute_rag_pipeline(row["question"])
>
>         # Use built-in RAGAS faithfulness evaluator
>         evaluation.evaluate(
>             "ragas/faithfulness",
>             index=index,
>             data={
>                 "input": row["question"],
>                 "output": response,
>                 "contexts": contexts,
>             },
>             settings={
>                 "model": "openai/gpt-6-luna",
>                 "max_tokens": 2048,
>             }
>         )
>
>         # Log custom metrics alongside
>         evaluation.log("confidence", index=index, score=response.confidence)
>
>     evaluation.submit(task, index, row)
> ```

> [tab] TypeScript
>
> Use `evaluation.evaluate()` with pre-built evaluators:
>
> ```typescript theme={null}
> await evaluation.run(dataset, async ({ item, index }) => {
>   const { response, contexts } = await executeRagPipeline(item.question);
>
>   // Use built-in RAGAS faithfulness evaluator
>   await evaluation.evaluate("ragas/faithfulness", {
>     index,
>     data: {
>       input: item.question,
>       output: response,
>       contexts,
>     },
>     settings: {
>       model: "openai/gpt-6-luna",
>       max_tokens: 2048,
>     },
>   });
>
>   // Log custom metrics alongside
>   evaluation.log("confidence", { index, score: response.confidence });
> });
> ```

> [!NOTE]
>
> Browse our complete list of [available evaluators](https://langwatch.ai/docs/evaluations/evaluators/list.md) including metrics
> for RAG quality, hallucination detection, safety, and more.

## Comparison

Scoring each target on its own leaves you calibrating numbers: is `0.78` better than `0.74` for your users, or is that noise? Comparison asks the easier question instead. It shows a judge every target output for the same row and asks which one is best.

`compare()` is the code-first way in. It uses the same judge as the [Comparison evaluator](https://langwatch.ai/docs/evaluations/experiments/ui/pairwise-compare.md) in the workbench, so a comparison you run from a notebook and one you set up in the UI read the same way on the results page.

### Comparing a row

Every target that produced a non-empty output for the row is a candidate, and one comparison weighs all of them together rather than a round of one-against-one matchups. Behind it the judge is called twice per row by default, the second time with the candidate order reversed, which is what [`swap_and_reconcile`](#options) does; turn that off and one comparison is one judge call.

There is no candidate list to assemble, and no judge prompt you have to choose, since the shipped one runs unless you [write your own](#customizing-the-judge-prompt):

> [tab] Python
>
> ```python theme={null}
> import langwatch
>
> evaluation = langwatch.experiment.init("model-comparison")
>
> for index, row in evaluation.loop(df.iterrows()):
>     with evaluation.target("gpt-6-luna", {"model": "openai/gpt-6-luna"}):
>         evaluation.log_response(ask_gpt(row["question"]))
>
>     with evaluation.target("claude-sonnet-5", {"model": "anthropic/claude-sonnet-5"}):
>         evaluation.log_response(ask_claude(row["question"]))
>
>     verdict = evaluation.compare(index, input=row["question"])
>     print(verdict.winner, verdict.reasoning)
> ```
>
> The candidates are whatever `log_response()` recorded inside each `target()` block.

> [tab] TypeScript
>
> ```typescript theme={null}
> import { LangWatch } from "langwatch";
>
> const langwatch = new LangWatch();
>
> const evaluation = await langwatch.experiments.init("model-comparison");
>
> await evaluation.run(dataset, async ({ item }) => {
>   await Promise.all([
>     evaluation.withTarget("gpt-6-luna", { model: "openai/gpt-6-luna" }, () =>
>       askGpt(item.question)
>     ),
>     evaluation.withTarget("claude-sonnet-5", { model: "anthropic/claude-sonnet-5" }, () =>
>       askClaude(item.question)
>     ),
>   ]);
>
>   const verdict = await evaluation.compare({ input: item.question });
>   console.log(verdict.winner, verdict.reasoning);
> });
> ```
>
> The candidates are whatever each `withTarget()` callback returned.

The winner comes back named with the target name you registered. `compare()` judges however many candidates the row has, and a target that produced no output for the row, or only an empty output, is not a candidate.

> [!NOTE]
>
> Compare a row once its targets have finished. In Python that is after the row's last `target()`
> block has closed, and in TypeScript after its `withTarget()` calls have settled, which under the
> usual `Promise.all` is right after the `await`.

> [!NOTE]
>
> `compare()` names the row the way `log()` does in the same SDK. Python is handed the index by `loop()`, so it passes it on as the first argument, and it needs a whole number. TypeScript infers it from the iteration the comparison runs in, so `index` is only needed when you compare outside a `run()` callback, and it is a whole number there too.
>
> Candidate order is shuffled per row by default, seeded by the row index, so re-comparing the same row presents the candidates in the same order and a different row shuffles differently. `randomize_order` / `randomizeOrder` governs it: turn it off and the judge sees the candidates in the order their targets were registered.

### Judging against a reference answer

By default the judge compares the candidates on their own merits, with no reference answer involved. Pass `golden` to have it judge each candidate against a reference instead. That one option is the whole switch, so there is no second flag to forget:

> [tab] Python
>
> ```python theme={null}
> verdict = evaluation.compare(
>     index,
>     input=row["question"],
>     golden=row["expected_answer"],
> )
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> const verdict = await evaluation.compare({
>   input: item.question,
>   golden: item.expectedAnswer,
> });
> ```

> [!TIP]
>
> Use a reference answer when correctness is the question. Leave it out for open-ended work, where
> tone, completeness, structure or policy adherence decide the winner and no single answer is the
> right one.

### Customizing the judge prompt

Leave `prompt` unset and the judge picks the shipped prompt matching what the row actually carries: a reference answer, task context, both, or neither. The framing for whatever the row lacks is dropped rather than left empty, which is why the default is worth keeping unless you have a rubric of your own.

When you do have one, pass it and it is used exactly as written. Three placeholders are filled in for you:

| Placeholder | Filled with |
| - | - |
| `{input}` | The `input` you passed, or an empty string when you passed none |
| `{golden}` | The `golden` you passed, or an empty string when you passed none |
| `{candidates}` | The candidate outputs, one per line, each behind a slot label |

> [tab] Python
>
> ```python theme={null}
> SUPPORT_RUBRIC = """\
> Pick the reply a support agent could send to the customer unedited.
>
> Question:
> {input}
>
> Candidates:
> {candidates}
>
> Weigh them in this order: factual accuracy first, then whether every part
> of the question is answered, then tone. Length is not quality. Explain
> briefly which candidate wins on those criteria, then pick its slot label.
> """
>
> verdict = evaluation.compare(
>     index,
>     input=row["question"],
>     prompt=SUPPORT_RUBRIC,
> )
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> const supportRubric = `Pick the reply a support agent could send to the customer unedited.
>
> Question:
> {input}
>
> Candidates:
> {candidates}
>
> Weigh them in this order: factual accuracy first, then whether every part
> of the question is answered, then tone. Length is not quality. Explain
> briefly which candidate wins on those criteria, then pick its slot label.`;
>
> const verdict = await evaluation.compare({
>   input: item.question,
>   prompt: supportRubric,
> });
> ```

> [!NOTE]
>
> Only those three placeholders are substituted, and they are substituted literally. Any other
> braces in your prompt reach the judge as written, so a rubric that says `score out of {10}` is
> passed through unchanged.

### Options

Every option is optional, because the judge already has a default for each one. An option you do not set is not sent at all, so the judge's default applies to it.

| Option | Default | What it does |
| - | - | - |
| `targets` | Every target that produced a non-empty output for the row | Restricts the comparison to the targets you name |
| `input` | No task context | The task the candidates were answering, for the judge to weigh them against |
| `golden` | No reference answer, so candidates are judged on their own merits | A reference answer to judge each candidate against, and the switch that turns reference judging on |
| `prompt` | The shipped prompt matching what the row carries | Your own judge prompt, used exactly as written |
| `model` | Your project's configured evaluator model | The judge model, for example `openai/gpt-6-luna` |
| `allow_tie` / `allowTie` | `true` | Lets the judge answer "tie" when no candidate is clearly better |
| `randomize_order` / `randomizeOrder` | `true` | Shuffles candidate order per row, seeded by the row index, so a candidate's position never sways the verdict |
| `swap_and_reconcile` / `swapAndReconcile` | `true` | Judges each row a second time with the candidate order reversed, and establishes no winner when the two passes disagree. Doubles the judge cost per row |
| `include_metrics` / `includeMetrics` | No metrics | Puts each candidate's `"duration"` in front of the judge, so it can prefer a faster candidate when quality is comparable. The time each target took is recorded for you. Cost is not on offer: the platform works a target's cost out from its traces after the run, so the SDK has none to show at the moment it asks for a verdict |
| `temperature` | `0` | The judge's sampling temperature. Models that only run at their own fixed temperature, the gpt-5 family among them, ignore it |
| `name` | `"comparison"` | The name the verdict is recorded under. Give each one its own name if you compare a row more than once |

### The verdict

A verdict carries four things: the `status`, the `winner`, the judge's `reasoning`, and `candidates`, the target names it actually judged.

| Status | What it means |
| - | - |
| `decided` | The judge picked a winner, named in `winner` |
| `tie` | The judge compared the candidates and found none better than the rest, so there is no winner |
| `inconclusive` | The judge ran and no winner came out of it. Under swap and reconcile that is usually its two passes disagreeing, so this row does not separate these candidates. That is a finding about them, and it is deliberately not a tie, which would claim they are equally good. It also covers a judge whose answer had no usable verdict in it, which `reasoning` says plainly. Either way the calls were made and billed, so the row still reports its cost |
| `skipped` | Fewer than two targets produced a non-empty output, so no judge ran. `reasoning` names the targets that produced no output |
| `error` | The judge failed or could not be reached, so the candidates were not measured at all |

> [!NOTE]
>
> A comparison never raises when a judge fails, and never raises on a row too thin to judge. One
> unreachable judge in a thousand-row run costs you that row, not the run.

> [!WARNING]
>
> Naming a target in `targets` that recorded no output for the row does raise. Comparing whatever
> remains would hand you a two-way verdict where you asked for a three-way one, which is worse than
> a failure.

### Seeing the results

Verdicts show up on the experiment's results page:

* A **Winner** column names the winning target on each row, with the judge's reasoning underneath it, so you can see why a row went the way it did.
* A **win-rate chart** gives each target its share of the wins, plus a bar for ties. A target that never won a single row is still charted, with zero wins.

With three or more targets there is also a leaderboard that ranks them from every matchup at once, with a confidence interval so the ranking's own uncertainty is visible. It is being rolled out gradually, so ask us to turn it on for your organization if you want it.

### A complete comparison

Three variants, one comparison per row, and a closing summary that fails a [CI build](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md) when the run had failures:

> [tab] Python
>
> ```python theme={null}
> import langwatch
>
> df = langwatch.dataset.get_dataset("your-dataset-id").to_pandas()
>
> evaluation = langwatch.experiment.init("support-reply-comparison")
>
> for index, row in evaluation.loop(df.iterrows(), threads=4):
>     def task(index, row):
>         with evaluation.target("gpt-6-luna", {"model": "openai/gpt-6-luna"}):
>             evaluation.log_response(ask_gpt(row["question"]))
>
>         with evaluation.target("claude-sonnet-5", {"model": "anthropic/claude-sonnet-5"}):
>             evaluation.log_response(ask_claude(row["question"]))
>
>         with evaluation.target("gemini-flash", {"model": "gemini/gemini-2.5-flash"}):
>             evaluation.log_response(ask_gemini(row["question"]))
>
>         verdict = evaluation.compare(
>             index,
>             input=row["question"],
>             golden=row["expected_answer"],
>             include_metrics=["duration"],
>         )
>         print(f"row {index}: {verdict.status} {verdict.winner or ''}")
>
>     evaluation.submit(task, index, row)
>
> evaluation.print_summary()
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> import { LangWatch } from 'langwatch';
>
> const langwatch = new LangWatch();
>
> const dataset = await loadDataset();
>
> const evaluation = await langwatch.experiments.init("support-reply-comparison");
>
> await evaluation.run(dataset, async ({ item, index }) => {
>   await Promise.all([
>     evaluation.withTarget("gpt-6-luna", { model: "openai/gpt-6-luna" }, () =>
>       askGpt(item.question)
>     ),
>     evaluation.withTarget("claude-sonnet-5", { model: "anthropic/claude-sonnet-5" }, () =>
>       askClaude(item.question)
>     ),
>     evaluation.withTarget("gemini-flash", { model: "gemini/gemini-2.5-flash" }, () =>
>       askGemini(item.question)
>     ),
>   ]);
>
>   const verdict = await evaluation.compare({
>     input: item.question,
>     golden: item.expectedAnswer,
>     includeMetrics: ["duration"],
>   });
>   console.log(`row ${index}: ${verdict.status} ${verdict.winner ?? ""}`);
> }, { concurrency: 4 });
>
> evaluation.printSummary();
> ```

### Comparing from an async loop

> [tab] Python
>
> In async-native mode, use `acompare()` so the judge is awaited rather than blocked on, leaving the event loop free for the rows running alongside it. It takes the same options and records the same verdict:
>
> ```python theme={null}
> import langwatch
>
> evaluation = langwatch.experiment.init("async-comparison")
>
> async def task(index, row):
>     with evaluation.target("gpt-6-luna"):
>         evaluation.log_response(await ask_gpt(row["question"]))
>
>     with evaluation.target("claude-sonnet-5"):
>         evaluation.log_response(await ask_claude(row["question"]))
>
>     verdict = await evaluation.acompare(index, input=row["question"])
>     print(verdict.winner)
>
> index = 0
> async for row in evaluation.aloop(dataset, concurrency=4):
>     evaluation.asubmit(task, index, row)
>     index += 1
> ```

> [tab] TypeScript
>
> `compare()` is already awaited, so the code stays the same: awaiting one row's verdict never holds up the rows running beside it. Raise `concurrency` and every row reaches the judge as soon as its own targets have settled.
>
> ```typescript theme={null}
> await evaluation.run(
>   dataset,
>   async ({ item }) => {
>     await Promise.all([
>       evaluation.withTarget("gpt-6-luna", () => askGpt(item.question)),
>       evaluation.withTarget("claude-sonnet-5", () => askClaude(item.question)),
>     ]);
>
>     const verdict = await evaluation.compare({ input: item.question });
>     console.log(verdict.winner);
>   },
>   { concurrency: 8 },
> );
> ```

> [!NOTE]
>
> The older `langevals/pairwise_compare` evaluator is deprecated. Existing calls keep working,
> because they are routed to the same judge described here, but new code should use `compare()`: it
> takes any number of candidates, names the winner with your own target name, and needs no candidate
> columns to assemble.

## Complete Example

> [tab] Python
>
> ```python theme={null}
> import langwatch
>
> # Load dataset from LangWatch
> df = langwatch.dataset.get_dataset("your-dataset-id").to_pandas()
>
> # Initialize evaluation
> evaluation = langwatch.experiment.init("rag-pipeline-evaluation-v2")
>
> # Run evaluation with parallelization
> for index, row in evaluation.loop(df.iterrows(), threads=8):
>     def task(index, row):
>         # Compare two RAG configurations
>         with evaluation.target("rag-v1", {"model": "gpt-6-luna", "retriever": "dense"}):
>             response, contexts = execute_rag_pipeline(row["question"], version="v1")
>             evaluation.log_response(response.text)  # Store the output
>
>             # Use LangWatch evaluators - target auto-inferred
>             evaluation.evaluate(
>                 "ragas/faithfulness",
>                 index=index,
>                 data={"input": row["question"], "output": response, "contexts": contexts},
>                 settings={"model": "openai/gpt-6-luna", "max_tokens": 2048}
>             )
>
>             # Log custom metrics - latency auto-captured by target()
>             evaluation.log("response_quality", index=index, score=response.quality)
>
>         with evaluation.target("rag-v2", {"model": "gpt-6-luna", "retriever": "hybrid"}):
>             response, contexts = execute_rag_pipeline(row["question"], version="v2")
>             evaluation.log_response(response.text)
>
>             evaluation.evaluate(
>                 "ragas/faithfulness",
>                 index=index,
>                 data={"input": row["question"], "output": response, "contexts": contexts},
>                 settings={"model": "openai/gpt-6-luna", "max_tokens": 2048}
>             )
>
>             evaluation.log("response_quality", index=index, score=response.quality)
>
>     evaluation.submit(task, index, row)
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> import { LangWatch } from 'langwatch';
>
> const langwatch = new LangWatch();
>
> // Your dataset (or load from LangWatch)
> const dataset = await loadDataset();
>
> // Initialize evaluation
> const evaluation = await langwatch.experiments.init("rag-pipeline-evaluation-v2");
>
> // Run evaluation with parallelization
> await evaluation.run(dataset, async ({ item, index }) => {
>   // Compare multiple RAG configurations in parallel
>   await Promise.all([
>     evaluation.withTarget("rag-v1", { model: "gpt-6-luna", retriever: "dense" }, async () => {
>       const { response, contexts } = await executeRagPipeline(item.question, "v1");
>
>       // Use LangWatch evaluators - target auto-inferred
>       await evaluation.evaluate("ragas/faithfulness", {
>         data: { input: item.question, output: response, contexts },
>         settings: { model: "openai/gpt-6-luna", max_tokens: 2048 }
>       });
>
>       // Log custom metrics - latency auto-captured by withTarget()
>       evaluation.log("response_quality", { score: response.quality });
>     }),
>
>     evaluation.withTarget("rag-v2", { model: "gpt-6-luna", retriever: "hybrid" }, async () => {
>       const { response, contexts } = await executeRagPipeline(item.question, "v2");
>
>       await evaluation.evaluate("ragas/faithfulness", {
>         data: { input: item.question, output: response, contexts },
>         settings: { model: "openai/gpt-6-luna", max_tokens: 2048 }
>       });
>
>       evaluation.log("response_quality", { score: response.quality });
>     }),
>   ]);
> }, { concurrency: 8 });
> ```

## Tracing Your Pipeline

To get complete visibility into your LLM pipeline, add tracing to your functions:

> [tab] Python
>
> ```python theme={null}
> import langwatch
>
> @langwatch.trace()
> def agent(question):
>     # Your RAG pipeline, chain, or agent logic
>     context = retrieve_documents(question)
>     completion = llm.generate(question, context)
>     return {"text": completion.text, "context": context}
>
> for index, row in evaluation.loop(df.iterrows()):
>     result = agent(row["question"])
>     evaluation.log("accuracy", index=index, score=0.9)
> ```
>
> > [!NOTE]
> >
> > Learn more in our [Python Integration Guide](https://langwatch.ai/docs/integration/python/guide.md).

> [tab] TypeScript
>
> ```typescript theme={null}
> import { getLangWatchTracer } from 'langwatch';
>
> const tracer = getLangWatchTracer('my-app');
>
> const agent = async (question: string) => {
>   return tracer.withActiveSpan('agent', async (span) => {
>     // Your RAG pipeline, chain, or agent logic
>     const context = await retrieveDocuments(question);
>     const completion = await llm.generate(question, context);
>     return { text: completion.text, context };
>   });
> };
>
> await evaluation.run(dataset, async ({ item, index }) => {
>   const result = await agent(item.question);
>   evaluation.log("accuracy", { index, score: 0.9 });
> });
> ```
>
> > [!NOTE]
> >
> > Learn more in our [TypeScript Integration Guide](https://langwatch.ai/docs/integration/typescript/guide.md).

With tracing enabled, you can click through from any evaluation result to see the complete execution trace, including all LLM calls, prompts, and intermediate steps.

## Exporting Results to CSV

After running your evaluations, you can export results to CSV for further analysis in spreadsheet tools like Excel or Google Sheets.

### How to Export

Click the **Export to CSV** button in the top-right corner of the evaluation results page to download a complete CSV file with all your data.

### CSV Structure

The exported CSV organizes the data by dataset rows and targets. Here's the complete column structure:

#### Row Index

| Column | Description |
| - | - |
| `index` | Row number (0-based) for cross-referencing with the UI |

#### Dataset Columns

All columns from your input dataset are included with their original names.

#### Target Columns (per target)

For each target in your evaluation, the following columns are exported:

| Column Pattern | Description | Example |
| - | - | - |
| `{target}_model` | Model used for this target | `gpt-6-luna_model` → `openai/gpt-6-luna` |
| `{target}_prompt_id` | Prompt configuration ID (for prompt targets) | `gpt-6-luna_prompt_id` → `prompt-abc123` |
| `{target}_prompt_version` | Prompt version number | `gpt-6-luna_prompt_version` → `2` |
| `{target}_{metadata_key}` | Custom metadata values | `gpt-6-luna_seed` → `42` |
| `{target}_output` | Model output (or individual output fields) | `gpt-6-luna_output` → `"The answer is 42"` |
| `{target}_cost` | Execution cost in USD | `gpt-6-luna_cost` → `0.0012` |
| `{target}_duration_ms` | Execution time in milliseconds | `gpt-6-luna_duration_ms` → `1250` |
| `{target}_error` | Error message if execution failed | `gpt-6-luna_error` → `"Rate limit exceeded"` |
| `{target}_trace_id` | Trace ID for viewing execution details | `gpt-6-luna_trace_id` → `trace_abc123` |

#### Evaluator Columns (per target, per evaluator)

For each evaluator applied to a target:

| Column Pattern | Description | Example |
| - | - | - |
| `{target}_{evaluator}_score` | Numeric score (0-1) | `gpt-6-luna_faithfulness_score` → `0.95` |
| `{target}_{evaluator}_passed` | Boolean pass/fail | `gpt-6-luna_faithfulness_passed` → `true` |
| `{target}_{evaluator}_label` | Classification label | `gpt-6-luna_sentiment_label` → `positive` |
| `{target}_{evaluator}_details` | Additional details or explanation | `gpt-6-luna_faithfulness_details` → `"All claims supported"` |
| `{target}_{evaluator}_cost` | Cost of running the evaluator | `gpt-6-luna_faithfulness_cost` → `0.0005` |
| `{target}_{evaluator}_duration_ms` | Evaluator execution time | `gpt-6-luna_faithfulness_duration_ms` → `850` |

#### Comparison Columns (per comparison)

A comparison grades the row as a whole rather than one target, so it gets its own block of columns after every target block, named after the comparison:

| Column Pattern | Description | Example |
| - | - | - |
| `{comparison}_winner` | Name of the winning target, `tie`, `no_verdict`, `unresolved`, or empty | `comparison_winner` → `gpt-6-luna` |
| `{comparison}_candidates` | The targets compared on this row, comma separated | `comparison_candidates` → `gpt-6-luna, claude-sonnet-5` |
| `{comparison}_reasoning` | The judge's explanation for the verdict | `comparison_reasoning` → `"gpt-6-luna gives the exact reset link"` |

What each [verdict](#the-verdict) writes into those three cells:

| Verdict | `{comparison}_winner` | `{comparison}_candidates` | `{comparison}_reasoning` |
| - | - | - | - |
| `decided` | The winning target's name | The targets judged on this row | The judge's explanation |
| `tie` | `tie` | The targets judged on this row | The judge's explanation |
| `inconclusive` | `no_verdict` | The targets judged on this row | Why the verdict did not hold |
| `skipped` | `no_verdict` | The candidates the row did have | Why the row could not be compared |
| `error` | Empty | Empty | Empty |

An empty winner is never a tie. The three cells go empty together for `error` and for a row the comparison never ran on, so `tie` in the winner column always means the judge saw the candidates and called the row even.

`no_verdict` is the shared export token for `inconclusive` and `skipped`. The two are separate in code, on the `verdict.status` your call returns, but they are recorded under one status, so a row read back from a completed run cannot be sorted into one or the other and naming either in the export would be a guess.

The reasoning cell explains why no verdict was recorded, either in the judge's own words or, when the judge itself never answered, in the evaluator's account of what came back. Keep the two apart by reading `verdict.status` at the point of comparison.

A row's candidates can be fewer than the comparison's full set of targets, because a target that produced no output for that row is left out of the matchup. The winner reads `unresolved` on the rare row where the judge's answer matches none of the candidates it was shown, and a winner naming a target the run no longer lists is written under its own name.

### Example CSV Output

For an evaluation comparing two targets, `gpt-6-luna` and `claude`, with a faithfulness evaluator:

```csv theme={null}
index,question,expected,gpt-6-luna_model,gpt-6-luna_output,gpt-6-luna_cost,gpt-6-luna_duration_ms,gpt-6-luna_faithfulness_score,gpt-6-luna_faithfulness_passed,claude_model,claude_output,claude_cost,claude_duration_ms,claude_faithfulness_score,claude_faithfulness_passed
0,What is 2+2?,4,openai/gpt-6-luna,The answer is 4,0.0012,1250,0.95,true,anthropic/claude-sonnet-5,2+2 equals 4,0.0008,980,0.92,true
1,Capital of France?,Paris,openai/gpt-6-luna,Paris is the capital of France,0.0015,1100,0.98,true,anthropic/claude-sonnet-5,The capital of France is Paris,0.0010,890,0.97,true
```

### Using the Data

The CSV export supports these analysis workflows:

> [accordion] Filter and compare models
>
> Use spreadsheet filters to compare specific models or configurations:
>
> * Filter by `{target}_model` to analyze specific model performance
> * Sort by `{target}_{evaluator}_score` to find best/worst performing samples
> * Filter by `{target}_error` to identify failed executions

> [accordion] Analyze costs and latency
>
> Calculate aggregate metrics across your evaluation: - Sum `{target}_cost` columns for total
> evaluation cost per model - Average `{target}_duration_ms` to compare response times - Identify
> outliers with high latency or cost

> [accordion] Group by metadata
>
> Analyze performance across different configurations: - Pivot tables by temperature, max\_tokens, or
> custom metadata - Compare prompt versions side-by-side - Track improvements across iterations

> [accordion] Debug failures
>
> Investigate problematic samples:
>
> * Filter rows where `{target}_error` is not empty
> * Cross-reference `index` with the UI for detailed inspection
> * Click through to traces using `{target}_trace_id`

> [!NOTE]
>
> All column headers are normalized to lowercase with spaces replaced by underscores for consistency
> and compatibility with data analysis tools.

## Running in CI/CD

You can run SDK experiments in your CI/CD pipeline. The `print_summary()` method outputs a structured summary and exits with code 1 if any evaluations fail:

```python theme={null}
import langwatch

experiment = langwatch.experiment.init("ci-quality-check")

for idx, row in experiment.loop(dataset.iterrows()):
    response = my_llm(row["input"])
    experiment.evaluate("ragas/faithfulness", index=idx, data={...})

# This will exit with code 1 if any evaluations failed
experiment.print_summary()
```

See [CI/CD Integration](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md) for complete examples with GitHub Actions, GitLab CI, and more.

## What's Next?

> [card] [CI/CD Integration](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md)
>
> Run experiments in your CI/CD pipeline

> [card] [View Evaluators](https://langwatch.ai/docs/evaluations/evaluators/list.md)
>
> Explore all available evaluation metrics

> [card] [Datasets](https://langwatch.ai/docs/datasets/overview.md)
>
> Learn about dataset management

> [card] [View Examples](https://langwatch.ai/docs/cookbooks/build-a-simple-rag-app.md)
>
> Check out example notebooks


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.