Evaluations Overview

Score your prompts and agents on a dataset before you ship, score production traffic as it arrives, and block unsafe responses before they reach the user.

Let your agent set this up. Copy the evaluations prompt into your coding agent to get started automatically.

An evaluation runs an evaluator on an input and an output and returns a verdict: a pass or fail, a score, a label, and the details behind it. LangWatch runs the same evaluators in three places: on a dataset in the Experiments Workbench, on production traces as an online evaluation, and inside your request as a guardrail.

Where evaluations run

Experiments

Online Evaluation

Guardrails

Evaluators

Which one to use

Goal	Use
Check a prompt change on a dataset before you deploy it	Experiments
Compare two prompts, models or agents on the same rows	Comparison
Gate a deployment on a pass rate from your pipeline	Run experiments from CI/CD
Score production traffic and follow the pass rate over time	Online Evaluation
Block a jailbreak or a policy violation before the response is sent	Guardrails
Get a Slack message or an email when a monitor fails	Online Evaluation with Automations
Score outputs by hand and collect the reviews	Annotations

Start

Platform

Open Experiments, click New Experiment, add a dataset, a prompt and an evaluator, then click Run. See Experiments.

Python

import langwatch

evaluation = langwatch.experiment.init("my-first-experiment")

for idx, row in evaluation.loop(dataset.iterrows()):
response = my_llm(row["input"])
evaluation.log("quality", index=idx, score=0.95)

TypeScript

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();
const evaluation = await langwatch.experiments.init("my-first-experiment");

await evaluation.run(dataset, async ({ item, index }) => {
  const response = await myLLM(item.input);
  evaluation.log("quality", { index, score: 0.95 });
});

To score production traffic, open Online Evaluations, click New Online Evaluation, pick an evaluator, set the trace filter and the sampling rate, and save. See Setting up monitors.

To block a response, call the evaluator from your code with as_guardrail=True and check passed before you answer the user:

import langwatch

@langwatch.trace()
def my_llm_call(user_input):
    guardrail = langwatch.evaluation.evaluate(
        "azure/jailbreak",
        name="Jailbreak Detection",
        as_guardrail=True,
        data={"input": user_input},
    )

    if not guardrail.passed:
        return "I can't help with that request."

    return call_llm(user_input)

See Guardrails for the TypeScript call and the error handling.

Related

Datasets

Annotations