Instant Evals targets and statements

What one judged row is, how the time window and the trace filter narrow the rows, the LangWatchQL statement each target writes for you, and how to write your own.

What is a target

A target is one of traces, threads or llm-spans. It sets what one judged row is and which text the judge reads for it. Pass it as --target on the CLI or target on the API; the default is traces.

Target	One row is	What the judge reads	Pick it for
traces	One trace	A digest of the trace’s spans, 8,000 tokens at most: failed spans, tool calls, model calls, then the slowest	Questions about what the agent did on one turn: tools called, errors, tests run
threads	One conversation	The whole thread with the agent’s work on each turn: model calls, tool calls with arguments and results, retrievals	Questions about how a conversation went or what the agent did across it: tone, resolution, tools called
llm-spans	One LLM call	The input and output messages of that call	Questions about one prompt: whether the model was asked two things at once, whether it followed the system prompt

A conversation is one row when its traces share a gen_ai.conversation.id. A project that does not set one has no rows under threads, so ask for traces instead.

The time window

langwatch instant-eval run "the customer sounds annoyed" --target threads --last 30d
langwatch instant-eval run "the customer sounds annoyed" --target threads \
  --start 2026-09-01T00:00:00Z --end 2026-09-08T00:00:00Z

--last takes a number and a unit, 30m, 24h, 7d or 2w, and defaults to 7d. --start and --end take an ISO 8601 timestamp or epoch milliseconds. The window is written into the statement as two bound instants, start_at and end_at, so a run that pages through its rows several times reads one fixed selection.

Narrowing the rows with a trace filter

langwatch instant-eval run "the agent called an external HTTP API" \
  --filter 'service:checkout AND cost:>0.01' --last 7d

--filter takes the same syntax as the Trace Explorer’s search bar, with AND, OR, NOT, parentheses, * wildcards on model, and ranges such as cost:>0.01 and duration:[100 TO 500]. Under threads a conversation is judged when any of its traces matches; under llm-spans a call is judged when its trace matches.

Group	Fields you can filter on
Identity	traceId, traceName, service, origin, user, customer, conversation, scenarioRun
Classification	topic, subtopic, label, model, status:error
Cost and size	cost, duration, tokens, promptTokens, completionTokens, tokensPerSecond, ttft, ttlt, spans, tokensEstimated
Prompts	selectedPrompt, lastUsedPrompt, promptVersion
Attributes	trace.attribute.<key>
Free text	A bare word, matched against the captured input, the captured output and the trace name

Fields that read spans, evaluations, events, annotations or scenario runs are refused by name, with the list above in the refusal, because dropping the condition would charge you for rows you meant to leave out. Ask for those with a statement of your own instead.

Group	Refused fields
Spans	spanId, spanName, spanType, spanStatus, rootSpanType, span.attribute.<key>, containsAi
Evaluations	eval, evaluator, evaluatorLabel, evaluatorPassed, evaluatorScore, evaluatorStatus, evaluatorVerdict, guardrail
Events and annotations	event, event.attribute.<key>, annotation, feedback
Scenario runs	scenario, scenarioSet, scenarioBatch, scenarioStatus, scenarioVerdict
Errors	error, errorMessage, status:ok, status:warning

The statement a target writes

Every run is one LangWatchQL statement. A target plus your questions expands into one, the run stores it, and run prints it under Statement: with the two window values under Parameters:. Each question becomes one column, named after the question.

traces, with one yes or no question:

SELECT
  TraceId,
  Attributes['gen_ai.conversation.id'] AS ThreadId,
  OccurredAt,
  eval(llm_readable_trace(TraceId, 8000), 'the customer sounds annoyed') AS q1
FROM analytics.traces
WHERE OccurredAt >= {start_at:DateTime}
  AND OccurredAt < {end_at:DateTime}
ORDER BY TraceId

threads, one row per conversation, addressed by the conversation’s last trace:

SELECT
  argMax(m.TraceId, m.OccurredAt) AS TraceId,
  m.ConversationId AS ThreadId,
  max(m.OccurredAt) AS OccurredAt,
  eval(llm_readable_thread(m.ConversationId, 18729), 'the customer sounds annoyed') AS q1
FROM analytics.trace_metrics AS m
WHERE m.OccurredAt >= {start_at:DateTime}
  AND m.OccurredAt < {end_at:DateTime}
  AND m.ConversationId != ''
GROUP BY m.ConversationId
ORDER BY ThreadId

llm-spans, one row per model call:

SELECT
  TraceId,
  SpanId,
  StartTime AS OccurredAt,
  eval(llm_messages_span(TraceId, SpanId), 'the model was asked to do two things at once') AS q1
FROM analytics.spans
WHERE StartTime >= {start_at:DateTime}
  AND StartTime < {end_at:DateTime}
  AND SpanAttributes['langwatch.span.type'] = 'llm'
ORDER BY TraceId, SpanId

A --filter on traces goes into that WHERE. On threads and llm-spans it goes into a subquery over analytics.traces with the same window, because the trace’s attributes are not columns of the conversation or span views.

What is the token budget per row

Under traces the digest is cut to 8,000 tokens. Under threads the budget handed to llm_readable_thread is the judge’s whole text budget, about 31,000 judge tokens once your questions are counted in, so an ordinary conversation arrives whole. The number in the statement, 18729 for one short question, is that budget in the renderer’s own count of four bytes per token. Under llm-spans the text goes to the judge whole, up to the same ceiling.

A long conversation costs more than a short one, and the estimate measures your rows as they are.

When a conversation is over the budget, you still get a verdict: every turn is shortened before any is dropped, and when turns do have to go, the opening and closing turns stay and a marker in the middle names the turns left out. A question about how the conversation ended still reads the ending.

threads reads the agent’s work because a question about tools needs it and a question about the conversation scores within 2.5 points of the plain transcript. To judge only what was said, at fewer tokens, write the statement with conversation_bounded, which shows each turn’s messages and the names of the tools it used:

eval(conversation_bounded(m.ConversationId, 8000, ''), 'the customer left angry') AS q1

Which thread function to use has the measurements behind the choice.

Writing your own statement

Take the statement a run printed, edit it, and send it back with --sql.

Run it before you judge it

Run the statement through langwatch query first to see the rows before you pay to judge them. It needs the two window values the run printed:

langwatch query "$(cat annoyed.sql)" \
  --param start_at="2026-09-11 00:00:00" \
  --param end_at="2026-09-18 00:00:00"

langwatch instant-eval run --sql-file annoyed.sql \
  --param start_at="2026-09-11 00:00:00" \
  --param end_at="2026-09-18 00:00:00" \
  --limit 10000

What a statement needs

A statement is a run when:

• it projects TraceId and at least one eval function column
• it projects SpanId too when it has one row per span, so you get one verdict per span
• it has an ORDER BY when you use a LIMIT, so every pass over the rows reads the same set

Send either --sql or --target, not both. Use --param only with your own statement.

Text functions

The text to judge is one of these functions over the row’s key:

Function	Text
conversation(ConversationId)	The thread as the chat view shows it: the system prompt once, then each turn’s user message and reply, with one line naming the tools the turn used. Best for questions about what was said
conversation_bounded(ConversationId, max_tokens, until_trace_id)	The same transcript under a token budget: every turn shortened before any is dropped, opening and closing turns kept. Pass '' as until_trace_id to read the whole thread, or a trace id to stop there, so a judgement reads only what the agent had seen
llm_readable_thread(ConversationId, max_tokens)	The thread with the agent’s work between each user message and reply: model calls with token counts, tool calls with arguments and results, retrievals. Best for questions about what the agent did, and the safe default
llm_readable_trace(TraceId, max_tokens)	The trace as a span digest, cut to the budget by keeping the structure and expanding the failed spans, then the tool calls, then the model calls, then the slowest
llm_messages(TraceId)	The chat messages of the trace’s main LLM call, as JSON {input, output}
llm_input_messages(TraceId)	The request side only: the history the model was given on this turn
llm_output_messages(TraceId)	The response side only: everything the agent did from the last request on, tool calls included
llm_messages_span(TraceId, SpanId)	The chat messages of one named span
trace_json(TraceId)	The whole trace as one JSON object, spans included

Eval functions

The question is one of these functions over that text:

Function	Answer
eval(text, 'question')	The probability the statement is true of the text, 0 to 1
eval_criteria(text, 'question', ['what counts as yes', 'what does not'])	The same, with the boundary spelled out
eval_passed(text, 'question', 0.7)	1 or 0 against the threshold you give
eval_score(text, 'question', 1, 5)	The probability-weighted mean on a whole-numbered scale of at most ten levels
eval_category(text, 'question', ['refund: wants money back', 'bug: reports something broken'])	The name of the most likely option
eval_category_probs(text, 'question', [...])	Every option’s probability, as a JSON object

The same functions work in a plain langwatch query, which judges the rows the query returns and charges for them the same way. A run is how you take one past the query’s page size, up to the row cap.