Saved Evaluators

Create reusable evaluator configurations on the platform and use them across experiments, monitors, and guardrails.

Saved evaluators are pre-configured evaluation setups that you create on the LangWatch platform. Once saved, you can reuse them in experiments, monitors, guardrails, or through the API without reconfiguring settings each time.

When to use Saved Evaluators:

• You want to reuse the same evaluation configuration across multiple places
• You prefer configuring evaluators via UI rather than code
• You want non-technical team members to create and manage evaluations
• You need consistent evaluation settings across your team

See also:

• Built-in Evaluators - Use evaluators directly without platform setup
• Custom Scoring - Send scores from your own evaluation logic

Creating a Saved Evaluator

Via the Platform UI

1. Go to Evaluators in your LangWatch project
2. Click New Evaluator
3. Select the evaluator type (e.g., LLM Boolean, PII Detection)
4. Configure the settings (model, prompt, thresholds, etc.)
5. Give it a descriptive name and save

Via the Evaluators Page

You can also manage saved evaluators from the dedicated Evaluators page at /{project}/evaluators.

Custom (Code) evaluators

A code evaluator is a saved evaluator whose logic is your own Python. Create one on the Evaluators page with New Evaluator → Custom (Code): declare its inputs (for example output and expected_output) and write a Code class whose __call__ takes those inputs and returns a dictionary with any of passed, score, label and details.

class Code:
    def __call__(self, output: str, expected_output: str):
        passed = expected_output.lower() in output.lower()
        return {"passed": passed, "score": 1.0 if passed else 0.0}

From code you can:

• Run it like any saved evaluator: POST /api/evaluations/evaluators/{slug}/evaluate, with data holding one value per declared input.
• Find it: GET /api/evaluators lists it with type: "code", and GET /api/evaluators/{idOrSlug} returns its code and inputs in config.
• Monitor with it: run it on incoming traces with POST /api/monitors, sending the evaluator’s id as evaluatorId and code/<evaluatorId> as checkType, plus mappings from trace fields to its inputs.

Creating a code evaluator and editing its code are done on the Evaluators page: POST /api/evaluators only creates built-in evaluator types.

Using Saved Evaluators

Saved evaluators are referenced using the evaluators/{slug} format, where {slug} is the unique identifier assigned when you create the evaluator.

Finding Your Evaluator Slug

1. Go to your saved evaluator on the platform
2. Click the ⋮ menu → Use via API
3. Copy the slug from the code examples

In Experiments

Python

import langwatch

df = langwatch.dataset.get_dataset("my-dataset").to_pandas()

experiment = langwatch.experiment.init("my-experiment")

for index, row in experiment.loop(df.iterrows()):
    output = my_llm(row["input"])

    # Use your saved evaluator
    experiment.evaluate(
        "evaluators/my-tone-checker",  # Your saved evaluator slug
        index=index,
        data={
            "input": row["input"],
            "output": output,
        },
    )

In Online Evaluation

Python

import langwatch

@langwatch.span()
def my_llm_step(user_input: str):
    output = my_llm(user_input)

    # Use your saved evaluator
    result = langwatch.evaluation.evaluate(
        "evaluators/my-tone-checker",  # Your saved evaluator slug
        name="Tone Check",
        data={
            "input": user_input,
            "output": output,
        },
    )

    return output

As Guardrails

Python

import langwatch

@langwatch.span()
def my_llm_step(user_input: str):
    # Use your saved evaluator as a guardrail
    guardrail = langwatch.evaluation.evaluate(
        "evaluators/my-safety-check",  # Your saved evaluator slug
        name="Safety Check",
        data={"input": user_input},
        as_guardrail=True,
    )

    if not guardrail.passed:
        return "I can't help with that request."

    return my_llm(user_input)

Via cURL

# Set your API key
API_KEY="$LANGWATCH_API_KEY"

# Call your saved evaluator
curl -X POST "https://app.langwatch.ai/api/v1/evaluations/evaluators/my-tone-checker/evaluate" \
     -H "X-Auth-Token: $API_KEY" \
     -H "Content-Type: application/json" \
     -d @- <<EOF
{
  "name": "Tone Check",
  "data": {
    "input": "your input text",
    "output": "your output text"
  }
}
EOF

Saved vs Built-in Evaluators

Aspect	Built-in Evaluators	Saved Evaluators
Slug format	provider/evaluator (e.g., ragas/faithfulness)	evaluators/{slug} (e.g., evaluators/my-checker)
Configuration	In code via settings parameter	Pre-configured on platform
Reusability	Copy settings across code	Reference by slug anywhere
Management	In codebase	In LangWatch platform UI
Team access	Developers only	Anyone with platform access

Best Practices

Naming Conventions

Use descriptive, consistent names for your saved evaluators:

• ✅ tone-checker-formal
• ✅ pii-detection-strict
• ✅ answer-quality-v2
• ❌ test1
• ❌ my-evaluator

When to Save an Evaluator

Save an evaluator when you:

• Use the same configuration in multiple places
• Want to manage settings from the UI
• Need non-developers to configure evaluations
• Want to version control evaluation criteria separately from code

Overriding Settings

You can override saved evaluator settings at runtime:

experiment.evaluate(
    "evaluators/my-llm-judge",
    index=index,
    data={...},
    settings={
        "model": "openai/gpt-5-mini",  # Override the saved model
    },
)

Next Steps

Built-in Evaluators

Custom Scoring

Evaluators List

Experiments