Annotation queues

Create a queue with reviewers and score types, send traces to it, work through the items one by one, correct a trace, and send the reviewed traces to a dataset.

An annotation queue is a list of traces waiting for review, with the people who review them and the score types they score with. A trace can also be sent to one person directly, without a queue.

Create a queue

Open Annotations and click + next to My queues. The Create Annotation Queue drawer asks for:

Field	What it is
Participants	The members who review this queue. At least one.
Name Annotation Queue	The queue name. The slug under it is used in the queue’s URL.
Description	What the reviewers should look for.
Score Type	The score types reviewers fill in. At least one. Add New creates a score type from here.

Click Save. The queue appears in the sidebar for every participant. Edit queue in the queue’s menu reopens the drawer.

Add traces to a queue

In the Trace Explorer, select traces and click Add to annotation queue in the action bar. From an open trace, the header menu has the same entry.

In Send to, pick one or more queues or people, then click Send. A trace is queued once per queue and once per person. From the inbox, select rows and click Add to queue; on a queue page the button reads Move to queue.

Send traces automatically

An automation with the Add to annotation queue action sends every trace that matches its filter to the queues and people you pick. Over the Triggers API, actionParams.annotators lists them as { "id", "name" }, where id is queue-<queueId> for a queue or user-<userId> for a person.

From code

Annotation queues have no public REST, CLI or MCP endpoints yet: creating a queue, adding items and working through them happen in the dashboard. Two things are available from code:

• The automation route above, to fill a queue from matching traces.
• Read-only queries with LangWatchQL: the annotation_queues, annotation_queue_items and annotation_queue_scores views, which is also where a queue’s id comes from.

The Annotations API reads and writes the annotations themselves.

Work the queue

Open a queue in the sidebar, or Inbox, and click a pending row. The review page shows the conversation with a bar at the bottom. Annotate on a reply opens the composer next to the conversation, and the checkbox on a turn marks that trace for the dataset step.

1. Annotate the messages with Annotate or Suggest, as on any trace. See Annotations.
2. Click Next. The item is marked completed and the next item opens. Previous goes back. The counter reads 1 of N.
3. On the last item the button reads Done. When no pending item is left for you, the page shows All tasks complete and the pending badge on Annotations in the navigation disappears.

An item is Pending until you click Next or Done on it, then Completed. The queue page lists the queue’s Members and keeps completed items under Status: Completed, with their scores, comment count and suggestion count.

Edit a trace

Edit trace in the bar, or in the trace header menu, opens the trace in Annotation mode. You can change the trace input, output and metadata, and each span’s name, type, input, output, parameters and error, or delete a span.

The bar counts the changes, for example 1 field changed or 3 fields changed, 1 span deleted, and names the scope of the edit, for example Trace output. An edited field is highlighted and shows its character delta, and Reset on it restores the original. Click Save, or Cancel to drop the draft.

The corrections are stored next to the original trace, never over it. Comments you write while editing are saved as you write them and are not part of the draft.

Send the reviewed traces to a dataset

Tick Add to dataset at the end in the bar. Every trace you annotate during the session is counted; the checkbox next to a turn adds or removes that trace. When you click Done on the last item, the Add to Dataset drawer opens with those traces.

Map the dataset’s annotations column to the annotations source to store the reviews with the row, and expected_output to the trace output. Where a suggestion exists, the dataset row takes the corrected text instead; the trace itself keeps its original output. See Datasets for the mapping.

From the inbox, Add to dataset in the row menu or in the selection bar opens the same drawer for the rows you pick.

Also check: Annotation scores to define the score types a queue uses.