LLM-as-a-Judge Boolean Evaluator

Use an LLM as a judge with a custom prompt to do a true/false boolean evaluation of the message.

POST /langevals/llm_boolean/evaluate

Try it

Python

Offline Evaluation

import langwatch

df = langwatch.dataset.get_dataset("dataset-id").to_pandas()

evaluation = langwatch.evaluation.init("my-incredible-evaluation")

for index, row in evaluation.loop(df.iterrows()):
    # your execution code here       
    evaluation.run(
        "langevals/llm_boolean",
        index=index,
        data={
            "input": row["input"],
            "output": output,
            "contexts": row["contexts"],
        },
        settings={}
    )

200

[
  {
    "status": "processed",
    "score": 123,
    "passed": true,
    "label": "<string>",
    "details": "<string>",
    "cost": {
      "currency": "<string>",
      "amount": 123
    },
    "raw_response": {},
    "error_type": "<string>",
    "traceback": [
      "<string>"
    ]
  }
]

Authorizations

X-Auth-Token · string · header · required

Body

application/json

data · object · required

Show child attributes

data. input · string

The input text to evaluate

data. output · string

The output text to evaluate

data. contexts · string[]

Context information for evaluation

settings · object

Evaluator settings

Show child attributes

settings. model · string · default: openai/gpt-5

The model to use for evaluation

settings. max_tokens · number · default: 8192

max_tokens setting

settings. prompt · string · default: "You are an LLM evaluator. We need the guarantee that the output answers what is being asked on the input

The system prompt to use for the LLM to run the evaluation

Response

200

application/json

Successful evaluation

status · enum<string> · required

Available options: processed, skipped, error

score · number

Evaluation score

passed · boolean

Whether the evaluation passed

label · string

Evaluation label

details · string

Additional details about the evaluation

cost · object

Show child attributes

cost. currency · string · required

cost. amount · number · required

raw_response · object

Raw response from the evaluator

error_type · string

Type of error if status is 'error'

traceback · string[]

Error traceback if status is 'error'