Fallback Chains

Survive provider outages, rate limits, and timeouts transparently.

A fallback chain is an ordered list of provider credentials the gateway tries in sequence when the primary fails. It’s per-virtual-key configuration; each VK owns its own chain.

Configuring a chain

On the VK edit screen:

1. Select a primary provider credential.
2. Add one or more fallback credentials in the order they should be tried.

The VK’s resolved configuration carries the chain and the attempt limit:

{
  "fallback": {
    "chain":        ["pc_primary_openai", "pc_anthropic", "pc_bedrock_us_east"],
    "max_attempts": 3
  }
}

chain is the ordered credential list. max_attempts (default 3) sets how many of them the gateway tries for one request. A key with no fallback gets exactly one attempt.

When fallback triggers

The gateway decides which failures move to the next provider. It reads the real upstream result, and the rule is the same for every virtual key. There is no per-key trigger list. A narrowed list could only turn a failure the gateway can recover from into one that your callers see.

Condition	Fires?	Why
Upstream 5xx	✅	Provider’s fault; next provider may work.
timeout	✅	Provider degraded.
429 rate_limit_exceeded	✅	Primary is throttled; secondary may have headroom.
network_error (DNS/TCP/TLS)	✅	Connectivity issue to primary.
Upstream 404 Not Found	✅	In a chain with more than one provider, this usually means that this provider does not serve that model. The next one can. Common with self-hosted and OpenAI-compatible providers.
circuit_breaker open	✅	Preemptive, gateway knows primary has been failing recently.
Upstream 400 Bad Request	❌	Client-fault. Surfacing the error is correct.
Upstream 401 Unauthorized	❌	Provider credential bad. Needs human fix, not masking.
Upstream 403 Forbidden	❌	Authorization issue. Silent switch would hide a real problem.
Client disconnect or an expired request deadline	❌	The caller went away. This tells nothing about the provider’s health.
LangWatch-internal invalid_api_key etc	❌	Never reaches the fallback layer.

Crossing providers mid-chain

A single VK may have mixed providers (OpenAI + Anthropic + Bedrock). The gateway uses Bifrost’s provider-dispatch library to translate payloads: same messages schema, different wire formats.

The model does not change across the chain. One request resolves to one model. The gateway sends that same model name to the next credential and expects the provider to serve it. So a chain helps when its entries overlap on the models you call: an Anthropic credential with a Bedrock credential that serves Anthropic models, or two OpenAI accounts in different regions.

The gateway skips a credential that cannot serve the resolved model. It does not call it. When the model name tells the gateway which provider owns it, through an alias, the provider/model form, or a known bare name, the chain visits only that provider’s credentials on the VK. If the name tells the gateway nothing, the gateway walks the chain as configured, and the providers’ own answers decide the outcome.

A VK narrowed to specific providers is enforced here too. If the model resolves to a provider that the key cannot use, the gateway refuses with model_not_allowed. The refusal names the model and tells the key’s owner to widen the key’s provider access.

Streaming

Fallback behaviour differs based on when failure occurs:

• Before first chunk emits → transparent fallback. The stream-setup call (bifrost.ChatCompletionStreamRequest) walks the chain the same way non-streaming dispatch does; the client sees a single stream from whichever slot accepted the request. X-LangWatch-Fallback-Count reports the skipped slot count.
• After first chunk has streamed → no mid-stream fallback. The gateway emits a terminal event: error frame (with code: upstream_mid_stream_failure) and closes the connection. The client may retry; a fresh request would then re-walk the chain.

This split is deliberate, splicing chunks from two providers would produce an inconsistent response with mismatched tool-call ids and accumulated-content replay. It’s a hard “no” per contract and enforced by a byte-exact assertion on the SSE error frame shape so future refactors can’t accidentally break it.

See Streaming → pre-connection fallback, mid-stream failure for the exact frame bytes and a worked example.

Observing the chain in traces

Fallback attribution today:

• Prometheus counter gateway_provider_attempts_total{credential_id, outcome} increments once per attempt, with outcome in success | fallback_success | retryable_5xx | not_found | rate_limit | timeout | network | circuit_open | non_retryable | chain_exhausted | context_done.
• Response header X-LangWatch-Fallback-Count: N, how many fallbacks were attempted before success.
• Request-id correlation via X-LangWatch-Gateway-Request-Id, join the metric + log line back to the specific trace.

Per-attempt nested spans (langwatch.fallback.attempt, .reason attrs) are a v1.1 observability follow-up. In v1, the counter + header give you the aggregate picture; per-attempt reasoning lives in the gateway log line for that request-id.

If all attempts in the chain fail, the gateway returns the last provider’s error envelope mapped to the OpenAI-compatible shape (provider_error or upstream_timeout as the type).

Circuit breaker

Each provider has an independent circuit breaker with a sliding window:

Default	Meaning
Window: 30 s	Failure events within the last 30 s count toward the open threshold
Threshold: 10 failures	10 failures in the window open the circuit
Open cooldown: 60 s	Circuit stays open for 60 s; skipped on new requests regardless of fallback order
Half-open probe: 1 request	After cooldown, a single probe is let through; if it succeeds, circuit closes; if it fails, another 60 s of open

Override at the service level:

Env var	Default
LW_GATEWAY_CIRCUIT_WINDOW_S	30
LW_GATEWAY_CIRCUIT_THRESHOLD	10
LW_GATEWAY_CIRCUIT_COOLDOWN_S	60

Per-replica, not shared: each gateway replica maintains its own breaker state. This is deliberate, under a large-scale outage, N replicas rediscovering the recovered provider independently is resilient; depending on Redis for breaker consensus is not.

Circuit state is emitted as a Prometheus gauge gateway_circuit_state{credential_id}, one series per credential slot. The state is encoded in the gauge value, not in a label: 0 closed, 1 open, 2 half-open. Alert on gateway_circuit_state == 1 for 5m to catch real provider outages (as distinct from transient blips).

When a request hits a circuit that’s currently open, the gateway skips to the next entry in the fallback chain immediately, no wasted round-trip to a provider we already know is down.

Sizing the chain

Returns decrease after 3 entries. Each extra attempt spends one more provider round-trip, one after the other. With the default max_attempts=3, the worst case is three upstream calls in a row. The client has usually stopped waiting before the last one answers. Use chains of 2 to 3 entries for latency-sensitive traffic. Lower max_attempts if you need a tighter total budget. Keep longer chains for batch and offline work, where wall-clock time is cheap.