Run Instant Evals on your own decision model

Run Instant Evals on a decision model in your own provider account, set up Azure, and read the rows your model cannot judge.

​

LangWatch Evals Preview or your own model

Instant Evals run on LangWatch Evals Preview by default, billed as in Limits and cost. You can pick a decision model in your own provider account instead, per organization, team or project. Your provider then bills you for each row, and LangWatch does not.

​

The models you can pick

The picker is one list: LangWatch’s Evals Preview first, then the models on your own keys. Each row shows its provider’s logo and a title. OpenAI’s and Azure’s gpt-6-luna both read Decisions API, and the logo tells them apart. The other titles name the provider and the model. A model whose provider has no key yet is listed but greyed out, with a line saying which key to add.

Provider to add	Model in the picker	What the provider needs
OpenAI	Decisions API	API key. Runs on the OpenAI Decisions API
Azure OpenAI	Decisions API	API key, endpoint and a gpt-6-luna deployment on your resource. See Azure
Azure OpenAI	Azure Microsoft Decision 1	API key, endpoint and a Microsoft Decision 1 deployment on Azure AI Foundry. See Azure
Inception	Inception Mercury Decide	API key
Perplexity	Perplexity Decider v1.2	API key
Perplexity	Perplexity Decider Flash v1.2	API key
Cloudflare Workers AI	Cloudflare Clef	Account id and API token
Cloudflare Workers AI	Cloudflare Clef Flash	Account id and API token
TypeSafe	TypeSafe Jev 1.14	API key
OpenRouter	OpenRouter Jev 1.14	API key
OpenRouter	OpenRouter Microsoft Decision 1	API key
OpenRouter	OpenRouter Liquid d1.1	API key
Fireworks	Fireworks Rune	API key

gpt-6-luna runs on the OpenAI Decisions API, so it needs that model on your OpenAI or Azure account. The other models need no OpenAI account.

​

Set it up

1. Add the provider

   In Settings > Model Providers, click Add Model Provider, pick the provider from the table and enter its key. Scope it to each organization, team or project that should use it. See Model Providers.

2. Pick the model

   In Default Models on the same page, add or edit a config and pick its scopes. In the Instant Evals picker, choose your model and save.

   
3. Run

   Start a run as usual. The new model judges the next rows within five seconds.

The config’s Instant Evals column then shows your model.

The picker offers a model only when every scope of the config holds an enabled key for it. An organization config sees only keys saved on the organization, not a key one project holds. A config on two projects sees only what both hold.

As for other default models, a project config beats a team config, which beats an organization config. A provider key saved on the project beats one on its team or organization.

​

Azure

Both Azure models use the Azure OpenAI provider, with its API key (AZURE_OPENAI_API_KEY) and endpoint (AZURE_OPENAI_ENDPOINT).

• Endpoint (AZURE_OPENAI_ENDPOINT): your resource URL, such as https://my-resource.openai.azure.com. LangWatch takes the resource name from it and calls https://my-resource.openai.azure.com/openai/v1/decisions or, for Microsoft Decision 1, https://my-resource.services.ai.azure.com/providers/microsoft/v1/systemone. Hosts on openai.azure.com, cognitiveservices.azure.com and services.ai.azure.com work. Other hosts, http, custom ports and API Management gateways are refused, so the key stays inside Azure.
• Decisions deployment (AZURE_OPENAI_DECISIONS_DEPLOYMENT): your gpt-6-luna deployment name. Leave empty if it is gpt-6-luna.
• Microsoft Decision 1 deployment (AZURE_DECISION_1_DEPLOYMENT): your Microsoft Decision 1 deployment name on Azure AI Foundry. Leave empty if it keeps the model’s name.

Keep Use API Gateway off. With it on, the endpoint and deployment fields are hidden and every row is skipped as classifier_key_invalid.

Deploy the model on your resource before you pick it, in the Azure portal, Azure AI Foundry or the Azure CLI:

az cognitiveservices account deployment create \
  --resource-group my-group --name my-resource \
  --deployment-name gpt-6-luna --model-name gpt-6-luna --model-format OpenAI --model-version 2026-09-22 \
  --sku-name GlobalStandard --sku-capacity 50

Without the deployment, every row is skipped as classifier_not_configured.

​

When your model cannot answer

A run on your own model never falls back to LangWatch Evals Preview. Rows it cannot judge are marked skipped with the reason in error, and the run carries on. List them with langwatch instant-eval results <run-id> --status skipped.

error	Why	Fix
classifier_key_invalid	The provider refused the key (401, 402 or 403), or the scope has no enabled provider row, no key, or an endpoint outside Azure	Check the row and its key at Settings > Model Providers
classifier_not_configured	The provider answered 404: the account does not hold the model, or the Azure deployment does not exist	Deploy the model, or set the deployment name on the Azure row
classifier_rate_limited	The provider answered 429 on every retry	Raise the rate limit or quota on your provider account
classifier_provider_unavailable	The provider answered 5xx on every retry	Try again later

Rate limits and outages are retried before the row is skipped. Refused keys and missing models are not.

​

What changes on your own model

• Cost. Your provider bills you. LangWatch does not, and the free budget does not apply.
• Limits. The row, question and text limits still apply.
• Data. Each row’s text and your questions go to your provider, under your agreement with it. LangWatch Evals Preview never sees them.
• Search bar. Still runs on LangWatch Evals Preview.