> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Online Evaluations: monitor your production traffic

> Create an online evaluation that scores every incoming trace or thread with an evaluator, filter and sample the traffic it runs on, and read the results on the trace and in analytics.

A monitor is an evaluator attached to your production traffic. Traces that match its preconditions and sampling rate are scored asynchronously, so the score never delays the response to your user. A Trace Level monitor scores each trace as it arrives; a Thread Level monitor waits for the thread to go idle first.

This guide sets up an LLM-as-a-Judge monitor named `Off-topic reply` that flags replies that drift away from the customer's question. The same steps apply to any evaluator, for example PII detection or jailbreak detection.

## Prerequisites

* Your application sends traces to LangWatch. See the [integration quick start](https://langwatch.ai/docs/integration/quick-start.md).
* For evaluators that call a model, a model provider configured in Settings.

## Create a monitor

**1. Pick an evaluator**

Open [Online Evaluations](https://app.langwatch.ai/@project/online-evaluations) and click **New Online Evaluation**. In the drawer, pick the **Evaluation Level**, then click **Select Evaluator**. Pick one of your saved evaluators, or click **New Evaluator** to open **Choose Evaluator Category**: **Expected Answer**, **LLM as Judge**, **RAG Quality**, **Quality Aspects** and **Safety**, plus **Custom (Code)** for a Python evaluator and **Custom (from Workflow)** for a workflow.

Pick **LLM as Judge**, then **LLM-as-a-Judge Boolean Evaluator**.

![The Choose Evaluator Category drawer](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/online-evaluation-category.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=5fba8d1bbc13111ad6619fd8cfc8ee30)

**2. Configure the evaluator**

Name the evaluator, pick the judge model and write the criteria in **Prompt**, for example: the reply passes only when it answers the customer's question and stays on the topic of the customer's order. A built-in evaluator such as PII Detection shows its own options here instead, for example the entity types to flag.

Under **Variables**, map the evaluator's fields to the trace. For an LLM-as-judge evaluator, `input` maps to **Full Trace (AI-Readable)** (`formatted_trace`) by default: every span of the trace, tool calls with their arguments and results included, so the judge can check what the agent did as well as what it replied. `output` maps to the trace output.

Other evaluators map `input` to the trace input and `output` to the trace output. Click **Create Evaluator**.

**3. Set the level, the filter and the sampling**

Back in the **New Online Evaluation** drawer, the evaluator is selected.

![The New Online Evaluation drawer with the evaluation level, the evaluator, the name, a precondition and the sampling rate](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/online-evaluation-drawer.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=66a9783afb75646d95a6d3e7873ced5a)

| Field | What it does |
| - | - |
| **Evaluation Level** | **Trace Level** scores each trace on its own as it arrives. **Thread Level** scores all traces of a thread together. See [Evaluate whole threads](#evaluate-whole-threads). |
| **Name** | The name shown in the list, on the trace and in analytics. |
| **Preconditions** | Optional. Click **Add Precondition** to run the monitor only on traces that match. Each rule reads a field from the trace, its metadata, its spans, its topics, earlier evaluations, events or annotations, and applies **contains**, **does not contain**, **matches regex** or **is**. With no precondition, the monitor runs on every trace. |
| **Sampling** | Optional. The share of matching traces to score, from `0.01` (1%) to `1.0` (100%). Lower it on expensive evaluators. |

Click **Create Online Evaluation**. The monitor is active as soon as it is created.

## Evaluate whole threads

A trace is one request and response. A thread is the whole conversation, grouped by the `thread_id` you send in the trace metadata. Some checks only make sense on the conversation: whether the user's problem was resolved, whether the assistant stayed consistent across turns, whether the tone drifted.

Pick **Thread Level** in the drawer. The evaluator's variables now map to the **Current Thread** source. An LLM-as-judge evaluator's `input` maps to **Full Thread (AI-Readable)** by default; other evaluators map it to the `traces` field, which holds every trace in the thread with its input and output.

The **Full Thread (AI-Readable)** field (`formatted_traces`) holds the thread as one text: each turn once, with the agent's work between the user message and the reply (model calls with token counts, tool calls with their arguments and results, retrievals). It is the same rendering as the [`llm_readable_thread`](https://langwatch.ai/docs/api-reference/query/extraction-functions#which-thread-function-to-use) query function.

The thread is rendered under half of the evaluator's max tokens setting: 64,000 tokens at the default of 128,000, less when the judge model's context window is smaller. A long thread shortens every turn before it drops one.

An LLM-as-judge evaluator always runs on long content. When the text is still longer than the evaluator's max tokens setting or the model's context window, the judge reads its opening and its ending with a marker naming how many tokens were left out of the middle, and the result details say so.

![The drawer with Thread Level selected and the Thread Idle Time select](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/online-evaluation-thread-level.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=fa77d2e0e0bb4d5c7024d0a639edf788)

**Thread Idle Time** sets how long the monitor waits after the last message before it scores the thread, from 1 to 30 minutes, 5 minutes by default. Without it the thread would be scored again on every new message. Pick **Disabled - evaluate on every trace** to score after each message anyway.

## Read the results

On a trace, the **Evals** section of the trace drawer lists each monitor that ran with its verdict, score and the reason it returned, and the header shows an **Eval** chip per monitor. The trace table has an **Evals** column too, so you can filter and sort the [Trace Explorer](https://app.langwatch.ai/@project/traces) by the monitor's result.

![The Evals section of a trace drawer with the Off-topic reply verdict](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/trace-evals.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=4eba5f7b7957acdbdde5006bd7699768)

The [Online Evaluations](https://app.langwatch.ai/@project/online-evaluations) list shows every monitor with its mode, its status and **Performance, last 7 days**: the pass rate or the average score with a trend against the previous week.

![The Online Evaluations list with the active Off-topic reply monitor and its 7 day pass rate](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/online-evaluations-list.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=2098847dc3719e7363ca9073d23803d9)

**View analytics** in the row menu opens the evaluations analytics page. At the top, **Overall Health** counts the evaluation runs and **Pass / Fail Trend** plots them over time across every monitor. Below, each monitor has a pass and fail share, a bar chart of passed and failed evaluations per day and a **Pass Rate Trend** line. A score evaluator shows its average score instead of a pass rate.

![The evaluations analytics page with the Off-topic reply monitor: the pass and fail share, the passed and failed evaluations per day and the Pass Rate Trend](https://mintcdn.com/langwatch/iG2HKiUWiJlQldW1/images/evaluations/online-evaluation-analytics.png?fit=max\&auto=format\&n=iG2HKiUWiJlQldW1\&q=85\&s=f8218f805b508bb7568c552efb58b4b8)

## Pause, edit and remove a monitor

The row menu on the list has **Edit**, **Disable** (or **Enable** on a paused monitor), **Replicate to another project** and **Delete**. A paused monitor keeps its history and stops scoring new traces.

## Act on the results

* To be notified when a monitor fails, create an automation on the evaluation result. See [Automations](https://langwatch.ai/docs/features/automations.md).
* To collect the flagged traces into a dataset, add an automation with the **Add To Dataset** action. See [Build datasets from traces](https://langwatch.ai/docs/datasets/automatically-from-traces.md).
* To block a request before the response reaches the user, run the evaluator as a guardrail from your code instead of as a monitor. See [Guardrails](https://langwatch.ai/docs/evaluations/guardrails/overview.md).

**Also check:** [Evaluators](https://langwatch.ai/docs/evaluations/evaluators/overview.md) for the full list of evaluators a monitor can run, and [Saved evaluators](https://langwatch.ai/docs/evaluations/evaluators/saved-evaluators.md) to reuse one evaluator across experiments and monitors.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.