> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Evaluations Overview

> Score your prompts and agents on a dataset before you ship, score production traffic as it arrives, and block unsafe responses before they reach the user.

> [!TIP]
>
> **Let your agent set this up.** [Copy the evaluations prompt](https://langwatch.ai/docs/skills/directory.md) into your coding agent to get started automatically.

An evaluation runs an evaluator on an input and an output and returns a verdict: a pass or fail, a score, a label, and the details behind it. LangWatch runs the same evaluators in three places: on a dataset in the Experiments Workbench, on production traces as an online evaluation, and inside your request as a guardrail.

## Where evaluations run

> [card] [Experiments](https://langwatch.ai/docs/evaluations/experiments/overview.md)
>
> Run a dataset through your prompts and agents in the workbench and compare runs before a change ships.

> [card] [Online Evaluation](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md)
>
> Attach an evaluator to your production traffic and read the pass rate over time.

> [card] [Guardrails](https://langwatch.ai/docs/evaluations/guardrails/overview.md)
>
> Call an evaluator from your code and block or change the response before the user sees it.

> [card] [Evaluators](https://langwatch.ai/docs/evaluations/evaluators/overview.md)
>
> The built-in evaluators, LLM judges with your own criteria, and custom code evaluators.

## Which one to use

| Goal | Use |
| - | - |
| Check a prompt change on a dataset before you deploy it | [Experiments](https://langwatch.ai/docs/evaluations/experiments/overview.md) |
| Compare two prompts, models or agents on the same rows | [Comparison](https://langwatch.ai/docs/evaluations/experiments/ui/pairwise-compare.md) |
| Gate a deployment on a pass rate from your pipeline | [Run experiments from CI/CD](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md) |
| Score production traffic and follow the pass rate over time | [Online Evaluation](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md) |
| Block a jailbreak or a policy violation before the response is sent | [Guardrails](https://langwatch.ai/docs/evaluations/guardrails/overview.md) |
| Get a Slack message or an email when a monitor fails | [Online Evaluation](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md) with [Automations](https://langwatch.ai/docs/features/automations.md) |
| Score outputs by hand and collect the reviews | [Annotations](https://langwatch.ai/docs/annotations/overview.md) |

## Start

> [tab] Platform
>
> Open [Experiments](https://app.langwatch.ai/@project/experiments), click **New Experiment**, add a dataset, a prompt and an evaluator, then click **Run**. See [Experiments](https://langwatch.ai/docs/evaluations/experiments/overview.md).

> [tab] Python
>
> ```python theme={null}
> import langwatch
>
> evaluation = langwatch.experiment.init("my-first-experiment")
>
> for idx, row in evaluation.loop(dataset.iterrows()):
>     response = my_llm(row["input"])
>     evaluation.log("quality", index=idx, score=0.95)
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> import { LangWatch } from "langwatch";
>
> const langwatch = new LangWatch();
> const evaluation = await langwatch.experiments.init("my-first-experiment");
>
> await evaluation.run(dataset, async ({ item, index }) => {
>   const response = await myLLM(item.input);
>   evaluation.log("quality", { index, score: 0.95 });
> });
> ```

To score production traffic, open [Online Evaluations](https://app.langwatch.ai/@project/online-evaluations), click **New Online Evaluation**, pick an evaluator, set the trace filter and the sampling rate, and save. See [Setting up monitors](https://langwatch.ai/docs/evaluations/online-evaluation/setup-monitors.md).

To block a response, call the evaluator from your code with `as_guardrail=True` and check `passed` before you answer the user:

```python theme={null}
import langwatch

@langwatch.trace()
def my_llm_call(user_input):
    guardrail = langwatch.evaluation.evaluate(
        "azure/jailbreak",
        name="Jailbreak Detection",
        as_guardrail=True,
        data={"input": user_input},
    )

    if not guardrail.passed:
        return "I can't help with that request."

    return call_llm(user_input)
```

See [Guardrails](https://langwatch.ai/docs/evaluations/guardrails/overview.md) for the TypeScript call and the error handling.

## Related

> [card] [Datasets](https://langwatch.ai/docs/datasets/overview.md)
>
> Create the datasets your experiments run on, by hand, from a CSV or from traces.

> [card] [Annotations](https://langwatch.ai/docs/annotations/overview.md)
>
> Review traces by hand with comments, scores and expected outputs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.