Experiments via SDK

Let your agent set this up. Copy the evaluations prompt into your coding agent to get started automatically.

The LangWatch SDK runs experiments from your Python or TypeScript code and tracks each run in LangWatch. Not using Python or TypeScript? The SDKs call the same REST endpoints you can call yourself: create an experiment, run it, then poll the run and read its results.

Quickstart

1. Install the SDK

Python
TypeScript

pip install langwatch

npm install langwatch
# or
pnpm add langwatch

2. Set your API Key

Python (Notebook)
Environment Variable

import langwatch

langwatch.login()

Be sure to log in or create an account using the displayed link, then provide your API key when prompted.

export LANGWATCH_API_KEY=your_api_key
export LANGWATCH_PROJECT_ID=your_project_id  # Required for service API keys

LANGWATCH_PROJECT_ID is required when using a service API key (e.g. for CI/CD or multi-project setups). Project API keys obtained via langwatch.login() or from the project settings page already have the project context built in.

3. Start tracking

Python
TypeScript

import langwatch
import pandas as pd

# Load your dataset
df = pd.read_csv("my_dataset.csv")

# Initialize a new experiment
evaluation = langwatch.experiment.init("my-experiment")

# Wrap your loop with evaluation.loop(), and iterate as usual
for idx, row in evaluation.loop(df.iterrows()):
    # Run your model or pipeline
    response = my_agent(row["question"])

    # Log a metric for this sample
    evaluation.log("sample_metric", index=idx, score=0.95)

import { LangWatch } from 'langwatch';

// Initialize the SDK
const langwatch = new LangWatch();

// Your dataset
const dataset = [
  { question: "What is 2+2?", expected: "4" },
  { question: "What is the capital of France?", expected: "Paris" },
];

// Initialize evaluation
const evaluation = await langwatch.experiments.init("my-experiment");

// Run evaluation with a callback
await evaluation.run(dataset, async ({ item, index }) => {
  // Run your model or pipeline
  const response = await myAgent(item.question);

  // Log a metric for this sample
  evaluation.log("sample_metric", { index, score: 0.95 });
});

Your evaluation metrics are now tracked and visualized in LangWatch.

Core Concepts

Evaluation Initialization

The evaluation is started by creating an evaluation session with a descriptive name:

Python
TypeScript

evaluation = langwatch.experiment.init("rag-pipeline-openai-vs-claude")

const evaluation = await langwatch.experiments.init("rag-pipeline-openai-vs-claude");

Iterating over data

Python
TypeScript

Use evaluation.loop() around your iterator so the entries are tracked:

for index, row in evaluation.loop(df.iterrows()):
    # Your existing evaluation code

Use evaluation.run() with a callback that receives each item:

await evaluation.run(dataset, async ({ item, index, span }) => {
  // Your existing evaluation code
});

The callback receives item (the current dataset item), index (the current index), and span (an OpenTelemetry span for custom tracing).

Metrics logging

Track any metric you want with evaluation.log():

Python
TypeScript

# Numeric scores
evaluation.log("relevance", index=index, score=0.85)

# Boolean pass/fail
evaluation.log("contains_citation", index=index, passed=True)

# Include additional data for debugging
evaluation.log("coherence", index=index, score=0.9,
               data={"output": result["text"], "tokens": result["token_count"]})

// Numeric scores
evaluation.log("relevance", { index, score: 0.85 });

// Boolean pass/fail
evaluation.log("contains_citation", { index, passed: true });

// Include additional data for debugging
evaluation.log("coherence", {
  index,
  score: 0.9,
  data: { output: result.text, tokens: result.tokenCount }
});

Comparing Multiple Targets

When comparing different models, prompts, or configurations, use targets to organize your results. Both SDKs provide a target(), withTarget() context that automatically captures latency and enables context inference. Once a row’s targets have each produced an output, Comparison judges them against each other and names a winner.

Python
TypeScript

Use evaluation.target() for automatic latency capture and context inference:

evaluation = langwatch.experiment.init("model-comparison")

for index, row in evaluation.loop(df.iterrows()):
    def compare_models(index, row):
        # Evaluate GPT-5 with automatic latency tracking
        with evaluation.target("gpt5-baseline", {"model": "openai/gpt-5"}):
            response = call_openai("gpt-5", row["question"])
            evaluation.log_response(response)  # Store the model output
            # Target is auto-inferred inside target()!
            evaluation.log("accuracy", index=index,
                          score=calculate_accuracy(response, row["expected"]))

        # Evaluate Claude with automatic latency tracking
        with evaluation.target("claude-experiment", {"model": "anthropic/claude-4-opus"}):
            response = call_anthropic("claude-4-opus", row["question"])
            evaluation.log_response(response)
            evaluation.log("accuracy", index=index,
                          score=calculate_accuracy(response, row["expected"]))

    evaluation.submit(compare_models, index, row)

evaluation.target() automatically captures latency, creates isolated traces per target, and enables context inference so log() calls don’t need explicit target parameters. Use log_response() to store the model’s output.

Alternatively, use the target parameter directly with evaluation.log():

evaluation.log(
    "accuracy",
    index=index,
    score=0.95,
    target="gpt5-baseline",
    metadata={"model": "openai/gpt-5", "temperature": 0.7}
)

Use withTarget() for automatic latency capture and context inference:

const evaluation = await langwatch.experiments.init("model-comparison");

await evaluation.run(dataset, async ({ item, index }) => {
  // Run targets in parallel with automatic tracing
  const [gpt5Result, claudeResult] = await Promise.all([
    evaluation.withTarget("gpt5-baseline", { model: "openai/gpt-5" }, async () => {
      const response = await callOpenAI("gpt-5", item.question);
      // Target and index are auto-inferred inside withTarget()!
      evaluation.log("accuracy", { score: calculateAccuracy(response, item.expected) });
      return response;
    }),

    evaluation.withTarget("claude-experiment", { model: "anthropic/claude-4-opus" }, async () => {
      const response = await callAnthropic("claude-4-opus", item.question);
      evaluation.log("accuracy", { score: calculateAccuracy(response, item.expected) });
      return response;
    }),
  ]);

  // Latency is automatically captured from each withTarget() span
  console.log(`GPT-5: ${gpt5Result.duration}ms, Claude: ${claudeResult.duration}ms`);
});

withTarget() automatically captures latency, creates isolated traces per target, and enables context inference so log() calls don’t need explicit target or index parameters.

Target Registration

The first time you use a target name, it’s automatically registered with the provided metadata:

Python
TypeScript

# Using target() - metadata is set when entering the context
with evaluation.target("gpt5", {"model": "gpt-5", "temp": 0.7}):
    evaluation.log_response("AI response here")  # Store the output
    evaluation.log("latency", index=0, score=150)  # target auto-inferred
    evaluation.log("accuracy", index=0, score=0.95)  # target auto-inferred

# Or using explicit target parameter (without target() context)
evaluation.log("latency", index=0, target="gpt5", metadata={"model": "gpt-5", "temp": 0.7})

# Subsequent calls can omit metadata - it's already registered
evaluation.log("accuracy", index=0, target="gpt5", score=0.95)
evaluation.log("latency", index=1, target="gpt5", score=150)

// Using withTarget() - metadata is set once when registering the target
await evaluation.withTarget("gpt5", { model: "gpt-5", temp: 0.7 }, async () => {
  evaluation.log("latency", { score: 150 });  // target auto-inferred
  evaluation.log("accuracy", { score: 0.95 }); // target auto-inferred
});

// Or using explicit target parameter
evaluation.log("latency", { index: 0, target: "gpt5", metadata: { model: "gpt-5", temp: 0.7 } });
evaluation.log("accuracy", { index: 0, target: "gpt5", score: 0.95 }); // metadata already registered

If you provide different metadata for the same target name, an error will be raised. Use a different target name if you want different configurations.

Metadata for Comparison

Target metadata is used for comparison charts in the LangWatch UI. You can group results by any metadata field:

Python
TypeScript

# Compare different temperatures
for temp in [0.0, 0.5, 0.7, 1.0]:
    for index, row in evaluation.loop(df.iterrows()):
        response = call_llm(row["question"], temperature=temp)
        evaluation.log(
            "quality",
            index=index,
            score=evaluate_quality(response),
            target=f"temp-{temp}",
            metadata={"model": "gpt-5", "temperature": temp}
        )

// Compare different temperatures
for (const temp of [0.0, 0.5, 0.7, 1.0]) {
  await evaluation.run(dataset, async ({ item, index }) => {
    const response = await callLLM(item.question, { temperature: temp });
    evaluation.log("quality", {
      index,
      score: evaluateQuality(response),
      target: `temp-${temp}`,
      metadata: { model: "gpt-5", temperature: temp }
    });
  });
}

In the LangWatch UI, you can then visualize how quality varies across temperature values.

Parallel Execution

LLM calls can be slow. Both SDKs support parallel execution to speed up your evaluations.

Python
TypeScript

Use the built-in parallelization by putting the content of the loop in a function and submitting it:

evaluation = langwatch.experiment.init("parallel-eval-example")

for index, row in evaluation.loop(df.iterrows(), threads=4):
    def task(index, row):
        result = agent(row["question"])  # Runs in parallel
        evaluation.log("response_quality", index=index, score=0.92)

    evaluation.submit(task, index, row)

By default, threads=4. Adjust based on your API rate limits and system resources.

Async-native mode

The default loop(), submit() path above already parallelises, each submitted task runs in a worker thread, so sync and async tasks both speed up with no extra work on your side. That’s the right choice for most users.Reach for aloop(), asubmit() only when your code is fully async-first and your task relies on async state whose identity is tied to one event loop. The threading path spins up a fresh event loop per worker, so those objects raise "Future attached to a different loop" on first use. aloop, asubmit keep every submitted task on the caller’s event loop, so that state stays valid across concurrent items.

evaluation = langwatch.experiment.init("async-eval-example")

async def task(index, row):
    result = await my_async_agent(row["question"])
    evaluation.log("response_quality", index=index, score=0.92)

index = 0
async for row in evaluation.aloop(dataset, concurrency=4):
    evaluation.asubmit(task, index, row)
    index += 1

Sync callables passed to asubmit are automatically offloaded to a worker thread so they don’t block the event loop for concurrent async siblings.

Pass the concurrency option to control how many items run in parallel:

await evaluation.run(dataset, async ({ item, index }) => {
  const result = await agent(item.question);  // Runs in parallel
  evaluation.log("response_quality", { index, score: 0.92 });
}, { concurrency: 4 });

By default, concurrency=4. Adjust based on your API rate limits and system resources.

Built-in Evaluators

LangWatch provides a library of evaluation metrics out of the box.

Python
TypeScript

Use evaluation.evaluate() with pre-built evaluators:

for index, row in evaluation.loop(df.iterrows()):
    def task(index, row):
        response, contexts = execute_rag_pipeline(row["question"])

        # Use built-in RAGAS faithfulness evaluator
        evaluation.evaluate(
            "ragas/faithfulness",
            index=index,
            data={
                "input": row["question"],
                "output": response,
                "contexts": contexts,
            },
            settings={
                "model": "openai/gpt-5",
                "max_tokens": 2048,
            }
        )

        # Log custom metrics alongside
        evaluation.log("confidence", index=index, score=response.confidence)

    evaluation.submit(task, index, row)

Use evaluation.evaluate() with pre-built evaluators:

await evaluation.run(dataset, async ({ item, index }) => {
  const { response, contexts } = await executeRagPipeline(item.question);

  // Use built-in RAGAS faithfulness evaluator
  await evaluation.evaluate("ragas/faithfulness", {
    index,
    data: {
      input: item.question,
      output: response,
      contexts,
    },
    settings: {
      model: "openai/gpt-5",
      max_tokens: 2048,
    }
  });

  // Log custom metrics alongside
  evaluation.log("confidence", { index, score: response.confidence });
});

Browse our complete list of available evaluators including metrics for RAG quality, hallucination detection, safety, and more.

Comparison

Scoring each target on its own leaves you calibrating numbers: is 0.78 better than 0.74 for your users, or is that noise? Comparison asks the easier question instead. It shows a judge every target output for the same row and asks which one is best. compare() is the code-first way in. It uses the same judge as the Comparison evaluator in the workbench, so a comparison you run from a notebook and one you set up in the UI read the same way on the results page.

Comparing a row

Every target that produced a non-empty output for the row is a candidate, and one comparison weighs all of them together rather than a round of one-against-one matchups. Behind it the judge is called twice per row by default, the second time with the candidate order reversed, which is what swap_and_reconcile does; turn that off and one comparison is one judge call. There is no candidate list to assemble, and no judge prompt you have to choose, since the shipped one runs unless you write your own:

Python
TypeScript

import langwatch

evaluation = langwatch.experiment.init("model-comparison")

for index, row in evaluation.loop(df.iterrows()):
    with evaluation.target("gpt-5-mini", {"model": "openai/gpt-5-mini"}):
        evaluation.log_response(ask_gpt(row["question"]))

    with evaluation.target("claude-sonnet-5", {"model": "anthropic/claude-sonnet-5"}):
        evaluation.log_response(ask_claude(row["question"]))

    verdict = evaluation.compare(index, input=row["question"])
    print(verdict.winner, verdict.reasoning)

The candidates are whatever log_response() recorded inside each target() block.

const evaluation = await langwatch.experiments.init("model-comparison");

await evaluation.run(dataset, async ({ item }) => {
  await Promise.all([
    evaluation.withTarget("gpt-5-mini", { model: "openai/gpt-5-mini" }, () =>
      askGpt(item.question)
    ),
    evaluation.withTarget("claude-sonnet-5", { model: "anthropic/claude-sonnet-5" }, () =>
      askClaude(item.question)
    ),
  ]);

  const verdict = await evaluation.compare({ input: item.question });
  console.log(verdict.winner, verdict.reasoning);
});

The candidates are whatever each withTarget() callback returned.

The winner comes back named with the target name you registered. compare() judges however many candidates the row has, and a target that produced nothing for the row, or only an empty output, is not a candidate.

Compare a row once its targets have finished. In Python that is after the row’s last target() block has closed, and in TypeScript after its withTarget() calls have settled, which under the usual Promise.all is right after the await.

compare() names the row the way log() does in the same SDK. Python is handed the index by loop(), so it passes it on as the first argument, and it needs a whole number. TypeScript infers it from the iteration the comparison runs in, so index is only needed when you compare outside a run() callback, and it is a whole number there too.Candidate order is shuffled per row by default, seeded by the row index, so re-comparing the same row presents the candidates in the same order and a different row shuffles differently. randomize_order / randomizeOrder governs it: turn it off and the judge sees the candidates in the order their targets were registered.

Judging against a reference answer

By default the judge compares the candidates on their own merits, with no reference answer involved. Pass golden to have it judge each candidate against a reference instead. That one option is the whole switch, so there is no second flag to forget:

Python
TypeScript

verdict = evaluation.compare(
    index,
    input=row["question"],
    golden=row["expected_answer"],
)

const verdict = await evaluation.compare({
  input: item.question,
  golden: item.expectedAnswer,
});

Use a reference answer when correctness is the question. Leave it out for open-ended work, where tone, completeness, structure or policy adherence decide the winner and no single answer is the right one.

Customizing the judge prompt

Leave prompt unset and the judge picks the shipped prompt matching what the row actually carries: a reference answer, task context, both, or neither. The framing for whatever the row lacks is dropped rather than left empty, which is why the default is worth keeping unless you have a rubric of your own. When you do have one, pass it and it is used exactly as written. Three placeholders are filled in for you:

Placeholder	Filled with
`{input}`	The `input` you passed, or nothing when you passed none
`{golden}`	The `golden` you passed, or nothing when you passed none
`{candidates}`	The candidate outputs, one per line, each behind a slot label

Python
TypeScript

SUPPORT_RUBRIC = """\
Pick the reply a support agent could send to the customer unedited.

Question:
{input}

Candidates:
{candidates}

Weigh them in this order: factual accuracy first, then whether every part
of the question is answered, then tone. Length is not quality. Explain
briefly which candidate wins on those criteria, then pick its slot label.
"""

verdict = evaluation.compare(
    index,
    input=row["question"],
    prompt=SUPPORT_RUBRIC,
)

const supportRubric = `Pick the reply a support agent could send to the customer unedited.

Question:
{input}

Candidates:
{candidates}

Weigh them in this order: factual accuracy first, then whether every part
of the question is answered, then tone. Length is not quality. Explain
briefly which candidate wins on those criteria, then pick its slot label.`;

const verdict = await evaluation.compare({
  input: item.question,
  prompt: supportRubric,
});

Only those three placeholders are substituted, and they are substituted literally. Any other braces in your prompt reach the judge as written, so a rubric that says score out of {10} is passed through unchanged.

Options

Every option is optional, because the judge already has a default for each one. An option you do not set is not sent at all, so the judge’s default applies to it.

Option	Default	What it does
`targets`	Every target that produced a non-empty output for the row	Restricts the comparison to the targets you name
`input`	No task context	The task the candidates were answering, for the judge to weigh them against
`golden`	No reference answer, so candidates are judged on their own merits	A reference answer to judge each candidate against, and the switch that turns reference judging on
`prompt`	The shipped prompt matching what the row carries	Your own judge prompt, used exactly as written
`model`	Your project’s configured evaluator model	The judge model, for example `openai/gpt-5-mini`
`allow_tie` / `allowTie`	`true`	Lets the judge answer “tie” when no candidate is clearly better
`randomize_order` / `randomizeOrder`	`true`	Shuffles candidate order per row, seeded by the row index, so a candidate’s position never sways the verdict
`swap_and_reconcile` / `swapAndReconcile`	`true`	Judges each row a second time with the candidate order reversed, and establishes no winner when the two passes disagree. Doubles the judge cost per row
`include_metrics` / `includeMetrics`	Nothing	Puts each candidate’s `"duration"` in front of the judge, so it can prefer a faster candidate when quality is comparable. The time each target took is recorded for you. Cost is not on offer: the platform works a target’s cost out from its traces after the run, so the SDK has none to show at the moment it asks for a verdict
`temperature`	`0`	The judge’s sampling temperature. Models that only run at their own fixed temperature, the gpt-5 family among them, ignore it
`name`	`"comparison"`	The name the verdict is recorded under. Give each one its own name if you compare a row more than once

The verdict

A verdict carries four things: the status, the winner, the judge’s reasoning, and candidates, the target names it actually judged.

Status	What it means
`decided`	The judge picked a winner, named in `winner`
`tie`	The judge compared the candidates and found none better than the rest, so there is no winner
`inconclusive`	The judge ran and no winner came out of it. Under swap and reconcile that is usually its two passes disagreeing, so this row does not separate these candidates. That is a finding about them, and it is deliberately not a tie, which would claim they are equally good. It also covers a judge that answered with nothing usable in it, which `reasoning` says plainly. Either way the calls were made and billed, so the row still reports its cost
`skipped`	Fewer than two targets produced a non-empty output, so no judge ran. `reasoning` names the targets that produced nothing
`error`	The judge failed or could not be reached, so nothing was measured about the candidates at all

A comparison never raises when a judge fails, and never raises on a row too thin to judge. One unreachable judge in a thousand-row run costs you that row, not the run.

Naming a target in targets that recorded no output for the row does raise. Comparing whatever remains would hand you a two-way verdict where you asked for a three-way one, which is worse than a failure.

Seeing the results

Verdicts show up on the experiment’s results page:

A Winner column names the winning target on each row, with the judge’s reasoning underneath it, so you can see why a row went the way it did.
A win-rate chart gives each target its share of the wins, plus a bar for ties. A target that never won a single row is still charted, with zero wins.

With three or more targets there is also a leaderboard that ranks them from every matchup at once, with a confidence interval so the ranking’s own uncertainty is visible. It is being rolled out gradually, so ask us to turn it on for your organization if you want it.

A complete comparison

Three variants, one comparison per row, and a closing summary that fails a CI build when the run had failures:

Python
TypeScript

import langwatch

df = langwatch.dataset.get_dataset("your-dataset-id").to_pandas()

evaluation = langwatch.experiment.init("support-reply-comparison")

for index, row in evaluation.loop(df.iterrows(), threads=4):
    def task(index, row):
        with evaluation.target("gpt-5-mini", {"model": "openai/gpt-5-mini"}):
            evaluation.log_response(ask_gpt(row["question"]))

        with evaluation.target("claude-sonnet-5", {"model": "anthropic/claude-sonnet-5"}):
            evaluation.log_response(ask_claude(row["question"]))

        with evaluation.target("gemini-flash", {"model": "gemini/gemini-2.5-flash"}):
            evaluation.log_response(ask_gemini(row["question"]))

        verdict = evaluation.compare(
            index,
            input=row["question"],
            golden=row["expected_answer"],
            include_metrics=["duration"],
        )
        print(f"row {index}: {verdict.status} {verdict.winner or ''}")

    evaluation.submit(task, index, row)

evaluation.print_summary()

import { LangWatch } from 'langwatch';

const langwatch = new LangWatch();

const dataset = await loadDataset();

const evaluation = await langwatch.experiments.init("support-reply-comparison");

await evaluation.run(dataset, async ({ item, index }) => {
  await Promise.all([
    evaluation.withTarget("gpt-5-mini", { model: "openai/gpt-5-mini" }, () =>
      askGpt(item.question)
    ),
    evaluation.withTarget("claude-sonnet-5", { model: "anthropic/claude-sonnet-5" }, () =>
      askClaude(item.question)
    ),
    evaluation.withTarget("gemini-flash", { model: "gemini/gemini-2.5-flash" }, () =>
      askGemini(item.question)
    ),
  ]);

  const verdict = await evaluation.compare({
    input: item.question,
    golden: item.expectedAnswer,
    includeMetrics: ["duration"],
  });
  console.log(`row ${index}: ${verdict.status} ${verdict.winner ?? ""}`);
}, { concurrency: 4 });

evaluation.printSummary();

Comparing from an async loop

Python
TypeScript

In async-native mode, use acompare() so the judge is awaited rather than blocked on, leaving the event loop free for the rows running alongside it. It takes the same options and records the same verdict:

evaluation = langwatch.experiment.init("async-comparison")

async def task(index, row):
    with evaluation.target("gpt-5-mini"):
        evaluation.log_response(await ask_gpt(row["question"]))

    with evaluation.target("claude-sonnet-5"):
        evaluation.log_response(await ask_claude(row["question"]))

    verdict = await evaluation.acompare(index, input=row["question"])
    print(verdict.winner)

index = 0
async for row in evaluation.aloop(dataset, concurrency=4):
    evaluation.asubmit(task, index, row)
    index += 1

compare() is already awaited, so nothing changes: awaiting one row’s verdict never holds up the rows running beside it. Raise concurrency and every row reaches the judge as soon as its own targets have settled.

await evaluation.run(dataset, async ({ item }) => {
  await Promise.all([
    evaluation.withTarget("gpt-5-mini", () => askGpt(item.question)),
    evaluation.withTarget("claude-sonnet-5", () => askClaude(item.question)),
  ]);

  const verdict = await evaluation.compare({ input: item.question });
  console.log(verdict.winner);
}, { concurrency: 8 });

The older langevals/pairwise_compare evaluator is deprecated. Existing calls keep working, because they are routed to the same judge described here, but new code should use compare(): it takes any number of candidates, names the winner with your own target name, and needs no candidate columns to assemble.

Complete Example

Python
TypeScript

import langwatch

# Load dataset from LangWatch
df = langwatch.dataset.get_dataset("your-dataset-id").to_pandas()

# Initialize evaluation
evaluation = langwatch.experiment.init("rag-pipeline-evaluation-v2")

# Run evaluation with parallelization
for index, row in evaluation.loop(df.iterrows(), threads=8):
    def task(index, row):
        # Compare two RAG configurations
        with evaluation.target("rag-v1", {"model": "gpt-5", "retriever": "dense"}):
            response, contexts = execute_rag_pipeline(row["question"], version="v1")
            evaluation.log_response(response.text)  # Store the output

            # Use LangWatch evaluators - target auto-inferred
            evaluation.evaluate(
                "ragas/faithfulness",
                index=index,
                data={"input": row["question"], "output": response, "contexts": contexts},
                settings={"model": "openai/gpt-5", "max_tokens": 2048}
            )

            # Log custom metrics - latency auto-captured by target()
            evaluation.log("response_quality", index=index, score=response.quality)

        with evaluation.target("rag-v2", {"model": "gpt-5", "retriever": "hybrid"}):
            response, contexts = execute_rag_pipeline(row["question"], version="v2")
            evaluation.log_response(response.text)

            evaluation.evaluate(
                "ragas/faithfulness",
                index=index,
                data={"input": row["question"], "output": response, "contexts": contexts},
                settings={"model": "openai/gpt-5", "max_tokens": 2048}
            )

            evaluation.log("response_quality", index=index, score=response.quality)

    evaluation.submit(task, index, row)

import { LangWatch } from 'langwatch';

const langwatch = new LangWatch();

// Your dataset (or load from LangWatch)
const dataset = await loadDataset();

// Initialize evaluation
const evaluation = await langwatch.experiments.init("rag-pipeline-evaluation-v2");

// Run evaluation with parallelization
await evaluation.run(dataset, async ({ item, index }) => {
  // Compare multiple RAG configurations in parallel
  await Promise.all([
    evaluation.withTarget("rag-v1", { model: "gpt-5", retriever: "dense" }, async () => {
      const { response, contexts } = await executeRagPipeline(item.question, "v1");

      // Use LangWatch evaluators - target auto-inferred
      await evaluation.evaluate("ragas/faithfulness", {
        data: { input: item.question, output: response, contexts },
        settings: { model: "openai/gpt-5", max_tokens: 2048 }
      });

      // Log custom metrics - latency auto-captured by withTarget()
      evaluation.log("response_quality", { score: response.quality });
    }),

    evaluation.withTarget("rag-v2", { model: "gpt-5", retriever: "hybrid" }, async () => {
      const { response, contexts } = await executeRagPipeline(item.question, "v2");

      await evaluation.evaluate("ragas/faithfulness", {
        data: { input: item.question, output: response, contexts },
        settings: { model: "openai/gpt-5", max_tokens: 2048 }
      });

      evaluation.log("response_quality", { score: response.quality });
    }),
  ]);
}, { concurrency: 8 });

Tracing Your Pipeline

To get complete visibility into your LLM pipeline, add tracing to your functions:

Python
TypeScript

@langwatch.trace()
def agent(question):
    # Your RAG pipeline, chain, or agent logic
    context = retrieve_documents(question)
    completion = llm.generate(question, context)
    return {"text": completion.text, "context": context}

for index, row in evaluation.loop(df.iterrows()):
    result = agent(row["question"])
    evaluation.log("accuracy", index=index, score=0.9)

Learn more in our Python Integration Guide.

import { getLangWatchTracer } from 'langwatch';

const tracer = getLangWatchTracer('my-app');

const agent = async (question: string) => {
  return tracer.withActiveSpan('agent', async (span) => {
    // Your RAG pipeline, chain, or agent logic
    const context = await retrieveDocuments(question);
    const completion = await llm.generate(question, context);
    return { text: completion.text, context };
  });
};

await evaluation.run(dataset, async ({ item, index }) => {
  const result = await agent(item.question);
  evaluation.log("accuracy", { index, score: 0.9 });
});

Learn more in our TypeScript Integration Guide.

With tracing enabled, you can click through from any evaluation result to see the complete execution trace, including all LLM calls, prompts, and intermediate steps.

Exporting Results to CSV

After running your evaluations, you can export results to CSV for further analysis in spreadsheet tools like Excel or Google Sheets.

How to Export

Click the Export to CSV button in the top-right corner of the evaluation results page to download a complete CSV file with all your data.

CSV Structure

The exported CSV organizes the data by dataset rows and targets. Here’s the complete column structure:

Row Index

Column	Description
`index`	Row number (0-based) for cross-referencing with the UI

Dataset Columns

All columns from your input dataset are included with their original names.

Target Columns (per target)

For each target in your evaluation, the following columns are exported:

Column Pattern	Description	Example
`{target}_model`	Model used for this target	`gpt-5-mini_model` → `openai/gpt-5-mini`
`{target}_prompt_id`	Prompt configuration ID (for prompt targets)	`gpt-5-mini_prompt_id` → `prompt-abc123`
`{target}_prompt_version`	Prompt version number	`gpt-5-mini_prompt_version` → `2`
`{target}_{metadata_key}`	Custom metadata values	`gpt-5-mini_seed` → `42`
`{target}_output`	Model output (or individual output fields)	`gpt-5-mini_output` → `"The answer is 42"`
`{target}_cost`	Execution cost in USD	`gpt-5-mini_cost` → `0.0012`
`{target}_duration_ms`	Execution time in milliseconds	`gpt-5-mini_duration_ms` → `1250`
`{target}_error`	Error message if execution failed	`gpt-5-mini_error` → `"Rate limit exceeded"`
`{target}_trace_id`	Trace ID for viewing execution details	`gpt-5-mini_trace_id` → `trace_abc123`

Evaluator Columns (per target, per evaluator)

For each evaluator applied to a target:

Column Pattern	Description	Example
`{target}_{evaluator}_score`	Numeric score (0-1)	`gpt-5-mini_faithfulness_score` → `0.95`
`{target}_{evaluator}_passed`	Boolean pass/fail	`gpt-5-mini_faithfulness_passed` → `true`
`{target}_{evaluator}_label`	Classification label	`gpt-5-mini_sentiment_label` → `positive`
`{target}_{evaluator}_details`	Additional details or explanation	`gpt-5-mini_faithfulness_details` → `"All claims supported"`
`{target}_{evaluator}_cost`	Cost of running the evaluator	`gpt-5-mini_faithfulness_cost` → `0.0005`
`{target}_{evaluator}_duration_ms`	Evaluator execution time	`gpt-5-mini_faithfulness_duration_ms` → `850`

Comparison Columns (per comparison)

A comparison grades the row as a whole rather than one target, so it gets its own block of columns after every target block, named after the comparison:

Column Pattern	Description	Example
`{comparison}_winner`	Name of the winning target, `tie`, `no_verdict`, `unresolved`, or empty	`comparison_winner` → `gpt-5-mini`
`{comparison}_candidates`	The targets compared on this row, comma separated	`comparison_candidates` → `gpt-5-mini, claude-sonnet-5`
`{comparison}_reasoning`	The judge’s explanation for the verdict	`comparison_reasoning` → `"gpt-5-mini gives the exact reset link"`

What each verdict writes into those three cells:

Verdict	`{comparison}_winner`	`{comparison}_candidates`	`{comparison}_reasoning`
`decided`	The winning target’s name	The targets judged on this row	The judge’s explanation
`tie`	`tie`	The targets judged on this row	The judge’s explanation
`inconclusive`	`no_verdict`	The targets judged on this row	Why the verdict did not hold
`skipped`	`no_verdict`	The candidates the row did have	Why the row could not be compared
`error`	Empty	Empty	Empty

An empty winner is never a tie. The three cells go empty together for error and for a row the comparison never ran on, so tie in the winner column always means the judge saw the candidates and called the row even. no_verdict is the shared export token for inconclusive and skipped. The two are separate in code, on the verdict.status your call returns, but they are recorded under one status, so a row read back from a completed run cannot be sorted into one or the other and naming either in the export would be a guess. The reasoning cell explains why no verdict was recorded, either in the judge’s own words or, when the judge itself never answered, in the evaluator’s account of what came back. Keep the two apart by reading verdict.status at the point of comparison. A row’s candidates can be fewer than the comparison’s full set of targets, because a target that produced no output for that row is left out of the matchup. The winner reads unresolved on the rare row where the judge’s answer matches none of the candidates it was shown, and a winner naming a target the run no longer lists is written under its own name.

Example CSV Output

For an evaluation comparing two targets, gpt-5-mini and claude, with a faithfulness evaluator:

index,question,expected,gpt-5-mini_model,gpt-5-mini_output,gpt-5-mini_cost,gpt-5-mini_duration_ms,gpt-5-mini_faithfulness_score,gpt-5-mini_faithfulness_passed,claude_model,claude_output,claude_cost,claude_duration_ms,claude_faithfulness_score,claude_faithfulness_passed
0,What is 2+2?,4,openai/gpt-5-mini,The answer is 4,0.0012,1250,0.95,true,anthropic/claude-sonnet-5,2+2 equals 4,0.0008,980,0.92,true
1,Capital of France?,Paris,openai/gpt-5-mini,Paris is the capital of France,0.0015,1100,0.98,true,anthropic/claude-sonnet-5,The capital of France is Paris,0.0010,890,0.97,true

Using the Data

The CSV export supports these analysis workflows:

Filter and compare models

Use spreadsheet filters to compare specific models or configurations:

Filter by {target}_model to analyze specific model performance
Sort by {target}_{evaluator}_score to find best/worst performing samples
Filter by {target}_error to identify failed executions

Analyze costs and latency

Calculate aggregate metrics across your evaluation:

Sum {target}_cost columns for total evaluation cost per model
Average {target}_duration_ms to compare response times
Identify outliers with high latency or cost

Group by metadata

Analyze performance across different configurations:

Pivot tables by temperature, max_tokens, or custom metadata
Compare prompt versions side-by-side
Track improvements across iterations

Debug failures

Investigate problematic samples:

Filter rows where {target}_error is not empty
Cross-reference index with the UI for detailed inspection
Click through to traces using {target}_trace_id

All column headers are normalized to lowercase with spaces replaced by underscores for consistency and compatibility with data analysis tools.

Running in CI/CD

You can run SDK experiments in your CI/CD pipeline. The print_summary() method outputs a structured summary and exits with code 1 if any evaluations fail:

import langwatch

experiment = langwatch.experiment.init("ci-quality-check")

for idx, row in experiment.loop(dataset.iterrows()):
    response = my_llm(row["input"])
    experiment.evaluate("ragas/faithfulness", index=idx, data={...})

# This will exit with code 1 if any evaluations failed
experiment.print_summary()

See CI/CD Integration for complete examples with GitHub Actions, GitLab CI, and more.

What’s Next?

CI/CD Integration

Run experiments in your CI/CD pipeline

View Evaluators

Explore all available evaluation metrics

Datasets

Learn about dataset management

View Examples

Check out example notebooks

​Quickstart

​1. Install the SDK

​2. Set your API Key

​3. Start tracking

​Core Concepts

​Evaluation Initialization

​Iterating over data

​Metrics logging

​Comparing Multiple Targets

​Target Registration

​Metadata for Comparison

​Parallel Execution

​Async-native mode

​Built-in Evaluators

​Comparison

​Comparing a row

​Judging against a reference answer

​Customizing the judge prompt

​Options

​The verdict

​Seeing the results

​A complete comparison

​Comparing from an async loop

​Complete Example

​Tracing Your Pipeline

​Exporting Results to CSV

​How to Export

​CSV Structure

​Row Index

​Dataset Columns

​Target Columns (per target)

​Evaluator Columns (per target, per evaluator)

​Comparison Columns (per comparison)

​Example CSV Output

​Using the Data

​Running in CI/CD

​What’s Next?

CI/CD Integration

View Evaluators

Datasets

View Examples

Quickstart

1. Install the SDK

2. Set your API Key

3. Start tracking

Core Concepts

Evaluation Initialization

Iterating over data

Metrics logging

Comparing Multiple Targets

Target Registration

Metadata for Comparison

Parallel Execution

Async-native mode

Built-in Evaluators

Comparison

Comparing a row

Judging against a reference answer

Customizing the judge prompt

Options

The verdict

Seeing the results

A complete comparison

Comparing from an async loop

Complete Example

Tracing Your Pipeline

Exporting Results to CSV

How to Export

CSV Structure

Row Index

Dataset Columns

Target Columns (per target)

Evaluator Columns (per target, per evaluator)

Comparison Columns (per comparison)

Example CSV Output

Using the Data

Running in CI/CD

What’s Next?