Automatically build datasets from real-time traces

Automatically build datasets from real-time traces to power LLM evaluations, regression tests, and AI agent testing workflows.

You can keep continuously populating the dataset with new data arriving from production by using Automations, mapping trace fields to any dataset columns you prefer.

Go to the Messages page and select a filter (for example, by model). The Add Automation button becomes enabled:

Add Automation

For Action, select Add To Dataset, and choose the right fields to map from the trace to the dataset:

Add To Dataset

Hit save. Every time a new message matches the filter, the automation fires and adds the new row to the dataset.