> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Online Evaluation Overview

> Score your production traffic as it arrives with monitors that run an evaluator on every matching trace or thread, and read the results on the trace, in the list and in analytics.

> [!TIP]
>
> **Let your agent set this up.** [Copy the evaluations prompt](https://langwatch.ai/docs/skills/directory.md) into your coding agent to get started automatically.

A monitor is an evaluator attached to your production traffic. It scores the traces your live application sends, so you see quality and safety problems on real traffic and collect the failing cases to fix. To score a dataset before you deploy a change, see [Experiments](https://langwatch.ai/docs/evaluations/experiments/overview.md).

## How it works

```text theme={null}
User request → Your application → Response → Trace in LangWatch → Monitor → Score
                                                                       ↓
                                                     Trace drawer, analytics, automations
```

1. Your application sends traces to LangWatch through an SDK or OpenTelemetry.
2. Each monitor checks its preconditions and its sampling rate against the incoming trace.
3. The monitor runs its evaluator on the trace, or on the whole thread once the thread goes idle.
4. The result lands on the trace and in the evaluations analytics, where an automation can pick it up.

The evaluator runs after the response is sent. A monitor never delays or blocks a response. To block a request, run the evaluator as a [guardrail](https://langwatch.ai/docs/evaluations/guardrails/overview.md) from your code.

## When to use online evaluation

| Use case | Example |
| - | - |
| Quality monitoring | An LLM-as-a-Judge with your criteria scores every conversation, and the pass rate trend shows a regression after a deploy. |
| Safety monitoring | PII detection, jailbreak detection or content moderation flags the traces that need a look. |
| Dataset building | An automation adds the traces a monitor failed to a dataset, so the next experiment runs on real failures. |
| Thread-level checks | A judge reads the whole conversation once it goes idle and checks whether the user's problem was resolved. |

## Monitors and guardrails

| Monitors | Guardrails |
| - | - |
| Score quality asynchronously | Block harmful content before it reaches the user |
| Run after the response is sent | Run inside your request, from your code |
| Feed the trace, analytics and automations | Return a pass or fail your code acts on |

## Create a monitor

1. Open [Online Evaluations](https://app.langwatch.ai/@project/online-evaluations) and click **New Online Evaluation**.
2. Pick an evaluator category and an evaluator, name it and map its inputs to the trace.
3. Pick **Trace Level** or **Thread Level**, add preconditions and a sampling rate if you need them, and click **Create Online Evaluation**.

The full walkthrough, with the thread-level options and where the results show, is in [Setting up Monitors](https://langwatch.ai/docs/evaluations/online-evaluation/setup-monitors.md).

## Add scores from your code

Your own code can record an evaluation result on the current span, next to the results of the monitors:

> [tab] Python
>
> ```python theme={null}
> import langwatch
>
> @langwatch.trace()
> def my_llm_app(user_input):
>     response = generate_response(user_input)
>
>     langwatch.get_current_span().add_evaluation(
>         name="response_quality",
>         passed=True,
>         score=0.95,
>         details="High quality response",
>     )
>
>     return response
> ```

> [tab] TypeScript
>
> ```typescript theme={null}
> import { getLangWatchTracer } from "langwatch";
>
> const tracer = getLangWatchTracer("my-app");
>
> await tracer.withActiveSpan("my_llm_app", async (span) => {
>   const response = await generateResponse(userInput);
>
>   span.addEvaluation({
>     name: "response_quality",
>     passed: true,
>     score: 0.95,
>     details: "High quality response",
>   });
>
>   return response;
> });
> ```

See [Custom scoring](https://langwatch.ai/docs/evaluations/evaluators/custom-scoring.md) for the result fields.

## Next steps

> [card] [Setting up Monitors](https://langwatch.ai/docs/evaluations/online-evaluation/setup-monitors.md)
>
> Create a monitor, filter and sample the traffic it scores, and read the results.

> [card] [Evaluators](https://langwatch.ai/docs/evaluations/evaluators/list.md)
>
> The evaluators a monitor can run, from PII detection to LLM-as-a-Judge.

> [card] [Automations](https://langwatch.ai/docs/features/automations.md)
>
> Get notified, or add traces to a dataset, when a monitor fails.

> [card] [Guardrails](https://langwatch.ai/docs/evaluations/guardrails/overview.md)
>
> Block harmful content before it reaches the user.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.