> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Evaluators Overview

> Understand evaluators - the scoring functions that assess your LLM outputs for quality, safety, and correctness.

> [!TIP]
>
> **Let your agent set this up.** [Copy the evaluations prompt](https://langwatch.ai/docs/skills/directory.md) into your coding agent to get started automatically.

Evaluators are scoring functions that assess the quality of your LLM's outputs. They're the building blocks for [experiments](https://langwatch.ai/docs/evaluations/experiments/overview.md), [online evaluation](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md), and [guardrails](https://langwatch.ai/docs/evaluations/guardrails/overview.md).

## Choose Your Approach

You can score an output in three ways:

> [card] [Built-in Evaluators](https://langwatch.ai/docs/evaluations/evaluators/built-in-evaluators.md)
>
> Use LangWatch's library of evaluators directly in your code.

> [card] [Saved Evaluators](https://langwatch.ai/docs/evaluations/evaluators/saved-evaluators.md)
>
> Create reusable evaluator configs on the platform.

> [card] [Custom Scoring](https://langwatch.ai/docs/evaluations/evaluators/custom-scoring.md)
>
> Send scores from your own evaluation logic.

### Which should I use?

| Approach | Slug Format | Best For |
| - | - | - |
| **Built-in Evaluators** | `provider/evaluator` (e.g., `ragas/faithfulness`) | Quick setup, standard evaluation methods |
| **Saved Evaluators** | `evaluators/{slug}` (e.g., `evaluators/my-checker`) | Team collaboration, UI-based configuration |
| **Custom Scoring** | N/A - you send the score directly | Proprietary logic, domain-specific metrics |

> [accordion] Decision flowchart
>
> ```text theme={null}
> Do you have your own evaluation logic?
> ├─ Yes → Use Custom Scoring
> └─ No → Do you want to configure via UI and reuse?
>          ├─ Yes → Use Saved Evaluators
>          └─ No → Use Built-in Evaluators
> ```

## What is an Evaluator?

An evaluator takes inputs (like the user question, LLM response, and optionally context or expected output) and returns a score indicating quality along some dimension.

```text theme={null}
Input + Output + Context → Evaluator → Score
                                        ↓
                              passed: true/false
                              score: 0.0 - 1.0
                              details: "explanation"
```

## Built-in Evaluator Categories

LangWatch provides a library of ready-to-use evaluators:

| Category | Examples | Use Case |
| - | - | - |
| **Expected Answer** | Exact Match, LLM Answer Match, LLM Factual Match, BLEU, ROUGE | Compare the output with a known answer |
| **LLM as Judge** | LLM-as-a-Judge Boolean, Score and Category, Rubrics Based Scoring | Apply your own criteria with a model |
| **RAG Quality** | Ragas Faithfulness, Ragas Response Relevancy, Context F1 | Evaluate retrieval-augmented generation |
| **Quality Aspects** | Valid Format, Language Detection, Sentiment, Query Resolution, SQL Query Equivalence | Check format, language and tone |
| **Safety** | PII Detection, Jailbreak Detection, Moderation, Off Topic, Competitor Blocklist | Detect harmful or unwanted content |

[Browse all evaluators →](https://langwatch.ai/docs/evaluations/evaluators/list.md)

The evaluator picker in the platform has two more entries. **Custom (Code)** runs a Python function you write in a sandbox with `requests`, `httpx`, `pydantic` and the LangWatch SDK available, with a 10 minute limit per call by default. **Custom (from Workflow)** runs one of your workflows as the evaluator.

## Quick Examples

### Using a Built-in Evaluator

```python theme={null}
import langwatch

# Use directly by slug
langwatch.evaluation.evaluate(
    "ragas/faithfulness",  # Built-in evaluator
    name="Faithfulness Check",
    data={
        "input": user_input,
        "output": response,
        "contexts": contexts,
    },
)
```

### Using a Saved Evaluator

```python theme={null}
import langwatch

# Use your saved evaluator by its slug
langwatch.evaluation.evaluate(
    "evaluators/my-tone-checker",  # Saved on platform
    name="Tone Check",
    data={
        "input": user_input,
        "output": response,
    },
)
```

### Sending Custom Scores

```python theme={null}
import langwatch

# Run your own logic and send the result
score = my_custom_evaluator(input, output)

langwatch.get_current_span().add_evaluation(
    name="my_custom_metric",
    passed=score > 0.7,
    score=score,
)
```

## Using Evaluators

### In Experiments

Run evaluators on each row of your test dataset for batch evaluation:

```python theme={null}
import langwatch

experiment = langwatch.experiment.init("my-experiment")

for idx, row in experiment.loop(df.iterrows()):
    response = my_llm(row["input"])

    experiment.evaluate(
        "ragas/faithfulness",
        index=idx,
        data={
            "input": row["input"],
            "output": response,
            "contexts": row["contexts"],
        },
    )
```

[Learn more about experiments →](https://langwatch.ai/docs/evaluations/experiments/overview.md)

### In Online Evaluation (Monitors)

Run evaluators automatically on production traces:

1. Create a monitor in LangWatch
2. Select evaluators to run
3. Configure when to trigger (all traces, sampled, filtered)
4. Scores appear on traces and dashboards

[Learn more about online evaluation →](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md)

### As Guardrails

Use evaluators to block harmful content in real-time:

```python theme={null}
import langwatch

guardrail = langwatch.evaluation.evaluate(
    "azure/jailbreak",
    name="Jailbreak Detection",
    data={"input": user_input},
    as_guardrail=True,
)

if not guardrail.passed:
    return "I can't help with that request."
```

[Learn more about guardrails →](https://langwatch.ai/docs/evaluations/guardrails/overview.md)

## Evaluator Inputs

Different evaluators require different inputs:

| Input | Description | Example Evaluators |
| - | - | - |
| `input` | User question/prompt | Jailbreak Detection, Off Topic Evaluator |
| `output` | LLM response | PII Detection, Valid Format Evaluator |
| `contexts` | Retrieved documents | Ragas Faithfulness, Ragas Response Context Precision |
| `expected_output` | Ground truth answer | LLM Answer Match, Exact Match Evaluator |
| `expected_contexts` | The documents that should have been retrieved | Context F1, Ragas Response Context Recall |

Check each evaluator's documentation for required and optional inputs.

## The `name` Parameter

> [!WARNING]
>
> **Important:** Always provide a descriptive `name` when running evaluators. This helps identify evaluation results in Analytics and traces.

```python theme={null}
import langwatch

# Good - descriptive name
langwatch.evaluation.evaluate(
    "langevals/llm_category",
    name="Answer Completeness Check",  # Descriptive!
    data={...},
)

# Bad - no name, hard to track
langwatch.evaluation.evaluate(
    "langevals/llm_category",
    data={...},
)
```

## Next Steps

> [card] [Built-in Evaluators](https://langwatch.ai/docs/evaluations/evaluators/built-in-evaluators.md)
>
> Use LangWatch's evaluator library directly.

> [card] [Saved Evaluators](https://langwatch.ai/docs/evaluations/evaluators/saved-evaluators.md)
>
> Create and reuse evaluator configurations.

> [card] [Custom Scoring](https://langwatch.ai/docs/evaluations/evaluators/custom-scoring.md)
>
> Send scores from your own evaluation logic.

> [card] [Evaluators List](https://langwatch.ai/docs/evaluations/evaluators/list.md)
>
> Browse all available evaluators.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.