Online Evaluations: set up a monitor on your production traffic

Create an online evaluation that scores every incoming trace or thread with an evaluator, filter and sample the traffic it runs on, and read the results on the trace and in analytics.

A monitor is an evaluator attached to your production traffic. Traces that match its preconditions and sampling rate are scored asynchronously, so the score never delays the response to your user. A Trace Level monitor scores each trace as it arrives; a Thread Level monitor waits for the thread to go idle first.

This guide sets up an LLM-as-a-Judge monitor named Off-topic reply that flags replies that drift away from the customer’s question. The same steps apply to any evaluator, for example PII detection or jailbreak detection.

Prerequisites

• Your application sends traces to LangWatch. See the integration quick start.
• For evaluators that call a model, a model provider configured in Settings.

Create a monitor

1. Pick an evaluator

   Open Online Evaluations and click New Online Evaluation. In the drawer, pick the Evaluation Level, then click Select Evaluator. Pick one of your saved evaluators, or click New Evaluator to open Choose Evaluator Category: Expected Answer, LLM as Judge, RAG Quality, Quality Aspects and Safety, plus Custom (Code) for a Python evaluator and Custom (from Workflow) for a workflow.

   Pick LLM as Judge, then LLM-as-a-Judge Boolean Evaluator.

   
2. Configure the evaluator

   Name the evaluator, pick the judge model and write the criteria in Prompt, for example: the reply passes only when it answers the customer’s question and stays on the topic of the customer’s order. A built-in evaluator such as PII Detection shows its own options here instead, for example the entity types to flag.

   Under Variables, map the evaluator’s fields to the trace. For an LLM-as-judge evaluator, input maps to Full Trace (AI-Readable) (formatted_trace) by default: every span of the trace, tool calls with their arguments and results included, so the judge can check what the agent did as well as what it replied. output maps to the trace output.

   Other evaluators map input to the trace input and output to the trace output. Click Create Evaluator.

3. Set the level, the filter and the sampling

   Back in the New Online Evaluation drawer, the evaluator is selected.

   

   Field	What it does
   Evaluation Level	Trace Level scores each trace on its own as it arrives. Thread Level scores all traces of a thread together. See Evaluate whole threads.
   Name	The name shown in the list, on the trace and in analytics.
   Preconditions	Optional. Click Add Precondition to run the monitor only on traces that match. Each rule reads a field from the trace, its metadata, its spans, its topics, earlier evaluations, events or annotations, and applies contains, does not contain, matches regex or is. With no precondition, the monitor runs on every trace.
   Sampling	Optional. The share of matching traces to score, from 0.01 (1%) to 1.0 (100%). Lower it on expensive evaluators.

   Click Create Online Evaluation. The monitor is active as soon as it is created.

Evaluate whole threads

A trace is one request and response. A thread is the whole conversation, grouped by the thread_id you send in the trace metadata. Some checks only make sense on the conversation: whether the user’s problem was resolved, whether the assistant stayed consistent across turns, whether the tone drifted.

Pick Thread Level in the drawer. The evaluator’s variables now map to the Current Thread source. An LLM-as-judge evaluator’s input maps to Full Thread (AI-Readable) by default; other evaluators map it to the traces field, which holds every trace in the thread with its input and output.

The Full Thread (AI-Readable) field (formatted_traces) holds the thread as one text: each turn once, with the agent’s work between the user message and the reply (model calls with token counts, tool calls with their arguments and results, retrievals). It is the same rendering as the llm_readable_thread query function.

The thread is rendered under half of the evaluator’s max tokens setting: 64,000 tokens at the default of 128,000, less when the judge model’s context window is smaller. A long thread shortens every turn before it drops one.

An LLM-as-judge evaluator always runs on long content. When the text is still longer than the evaluator’s max tokens setting or the model’s context window, the judge reads its opening and its ending with a marker naming how many tokens were left out of the middle, and the result details say so.

Thread Idle Time sets how long the monitor waits after the last message before it scores the thread, from 1 to 30 minutes, 5 minutes by default. Without it the thread would be scored again on every new message. Pick Disabled - evaluate on every trace to score after each message anyway.

Read the results

On a trace, the Evals section of the trace drawer lists each monitor that ran with its verdict, score and the reason it returned, and the header shows an Eval chip per monitor. The trace table has an Evals column too, so you can filter and sort the Trace Explorer by the monitor’s result.

The Online Evaluations list shows every monitor with its mode, its status and Performance, last 7 days: the pass rate or the average score with a trend against the previous week.

View analytics in the row menu opens the evaluations analytics page. At the top, Overall Health counts the evaluation runs and Pass / Fail Trend plots them over time across every monitor. Below, each monitor has a pass and fail share, a bar chart of passed and failed evaluations per day and a Pass Rate Trend line. A score evaluator shows its average score instead of a pass rate.

Pause, edit and remove a monitor

The row menu on the list has Edit, Disable (or Enable on a paused monitor), Replicate to another project and Delete. A paused monitor keeps its history and stops scoring new traces.

Act on the results

• To be notified when a monitor fails, create an automation on the evaluation result. See Automations.
• To collect the flagged traces into a dataset, add an automation with the Add To Dataset action. See Build datasets from traces.
• To block a request before the response reaches the user, run the evaluator as a guardrail from your code instead of as a monitor. See Guardrails.

Also check: Evaluators for the full list of evaluators a monitor can run, and Saved evaluators to reuse one evaluator across experiments and monitors.