Realtime voice

Mint a vendor session credential on a virtual key, so voice spend lands under a budget while the media socket runs client to vendor.

How the broker works

The gateway mints the vendor’s own session credential. Your client asks the gateway for a session, the gateway admits it against the key’s budgets and caps, calls the vendor’s mint endpoint with your stored provider key, and hands back what the vendor answered plus a LangWatch session id.

The audio never passes through the gateway. The media socket runs from your client straight to the vendor, so the gateway does not add to turn latency and cannot see, record or end a call in progress.

A session is admitted at the mint. Its spend is recorded as usage arrives: one record per report your client posts, or one at the end when the vendor reports the call.

Because the gateway is not in the audio path, a voice budget admits at session start and reconciles when the call ends. Overshoot on voice is bounded by the length of a call.

End users

Send X-LangWatch-End-User-Id on the request that creates the session (the mint, or the session create of a gateway-metered call). X-Litellm-End-User-Id is accepted as an alias. The gateway stores the id on the session, so every usage report, the closing record and an estimated settlement are attributed to that end user and count against a per-end-user budget. A usage report does not take the header: the session’s end user is fixed when it is created.

A session request has no user body field, so the header is the only source. On a key with an active per-end-user budget, a session request without the header answers 400 end_user_required. When a report takes that end user’s allowance to its limit, the usage response answers budget.exceeded: true with scope attributed_user.

ElevenLabs Conversational AI

Mint a signed URL for one hosted agent.

curl "https://gateway.langwatch.ai/v1/convai/conversation/get-signed-url?agent_id=agent_123" \
  -H "xi-api-key: $LANGWATCH_VIRTUAL_KEY"

The path and the header are ElevenLabs’ own, so a client that already talks to ElevenLabs changes only its base URL and its key. agent_id is required: a signed URL is bound to one agent, and a request without it answers 400 bad_request.

The answer is the vendor’s own body with one added key:

{
  "signed_url": "wss://api.elevenlabs.io/v1/convai/conversation?agent_id=agent_123&conversation_signature=...&conversation_id=conv_456",
  "langwatch": { "session_id": "req_01K6Y3M8QF7W2VZ4N9DXTB5C0A" }
}

The same id is on the X-LangWatch-Session-Id response header, so a client that parses the vendor shape strictly reads it there instead.

Open the socket with signed_url exactly as returned. The gateway asks ElevenLabs for the conversation id at mint time, so the call is joined to its spend record before the socket exists.

Closing an ElevenLabs session

ElevenLabs reports cost and duration after the call, not over the socket. LangWatch reads that report two ways, and either one settles the session:

1. The post-call webhook. Point your ElevenLabs workspace webhook at https://gateway.langwatch.ai/v1/convai/webhook/<model-provider-id> and store the webhook secret on that provider row. See ElevenLabs for the route, the signature check and the status codes.
2. The poller. Two minutes after a mint, LangWatch reads the conversation back from ElevenLabs by its own id.

A session neither of them closes settles as cost unknown, and a report arriving later replaces that record.

Only the post_call_transcription event is applied. post_call_audio and call_initiation_failure are acknowledged and not applied: they name the conversation and carry no duration.

ElevenLabs speech and transcription sockets

POST /v1/single-use-token/{token_type} mints a token that opens one ElevenLabs speech or transcription socket. See ElevenLabs for the mint and for how each socket takes the token. Usage is reported and the session is closed with the same two routes as an OpenAI session, described below.

OpenAI Realtime

Mint an ephemeral client secret.

curl -X POST "https://gateway.langwatch.ai/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"session":{"type":"realtime","model":"gpt-realtime-2.1"},
       "expires_after":{"anchor":"created_at","seconds":600}}'

This is OpenAI’s own path and its own body. The gateway reads session.model as the model and forwards your declaration as you wrote it, with two edits: the resolved model is written back into session.model, and expires_after.seconds is held between 10 and 7200. A body that names no expiry is forwarded untouched, so OpenAI’s own default applies. The voice, the tools, the turn detection and the audio formats reach OpenAI unchanged. The request body is capped at 256 KiB.

The answer is OpenAI’s own body plus langwatch.session_id, and the same id is on X-LangWatch-Session-Id.

If the session declares input transcription at session.audio.input.transcription.model, the gateway records that model at the mint. Transcription usage you report later is priced under it.

Open the socket with the returned secret. Which form you use depends on where the code runs, because a browser WebSocket takes only a URL and a subprotocol list and cannot set request headers.

From a server, with the ws package or any Node websocket client that accepts headers:

import WebSocket from "ws";

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1", {
  headers: { Authorization: `Bearer ${clientSecret.value}` },
});

From a browser, as a subprotocol:

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1", [
  "realtime",
  `openai-insecure-api-key.${clientSecret.value}`,
]);

Reporting OpenAI usage

OpenAI reports usage over the socket, and that socket does not pass through the gateway. Post what you read back to the session. SESSION_ID is the X-LangWatch-Session-Id from the mint.

Forward each response.done event as it arrives:

curl -X POST "https://gateway.langwatch.ai/v1/realtime/sessions/$SESSION_ID/usage" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"type":"response.done","response":{"id":"resp_001",
       "usage":{"input_tokens":132,"output_tokens":121,
         "input_token_details":{"text_tokens":119,"audio_tokens":13,"cached_tokens":64,
           "cached_tokens_details":{"text_tokens":64,"audio_tokens":0}},
         "output_token_details":{"text_tokens":30,"audio_tokens":91}}}}'

{
  "session_id": "req_01K6Y3M8QF7W2VZ4N9DXTB5C0A",
  "status": "recorded",
  "cost_usd": 0.007206,
  "session_cost_usd": 0.0412,
  "budget": { "exceeded": false }
}

Each report is its own spend record, so the key’s budgets see the call while it runs. The session stays open until you close it.

A report key is stored as letters, digits, _ and -, up to 128 characters. OpenAI’s response and item ids fit as they are.

Body	Report key	What it does
A whole response.done event	response.id	Records that response. Sending it again is not recorded twice.
A whole conversation.item.input_audio_transcription.completed event	item_id	Records input transcription, priced under the session’s transcription model. Its usage may be {"type":"tokens", ...} or {"type":"duration","seconds":N}. An event with no usage answers 200 with status no_usage.
{"events":[ ... ]}	One per event	A batch of the two events above, at most 100, recorded in order. A response.done in a batch must carry response.id.
A usage object, or {"usage":{...}}, with no id	None	The session total. Closes the session. If keyed reports came first, only the part they did not cover is charged.
{"id":"...","usage":{...}}	Your id	Records one report under an id you choose.

Add "final": true at the top level of any body to close the session after it is recorded. Add "duration_ms" beside it to state how long the call ran. A usage object with no id and "final": true is recorded as one more amount on top of the earlier reports, not as the session total.

Only ids and usage counts are read. Transcripts and prompts on a forwarded event are not looked at. The body is capped at 1 MiB.

Audio tokens are taken out of the text totals before pricing, because audio tokens cost several times what text tokens cost. When the usage carries cached_tokens_details, cached text tokens are priced at the cache-read rate and cached audio tokens are priced at the full audio input rate.

The usage response

Every usage post and every close answers 200 with this body:

Field	Meaning
session_id	The session the report was recorded against.
status	recorded, duplicate (that report key was already recorded), closed (the report closed the session), already_closed (the session was closed before and this post was not recorded), or no_usage (the body carried no usage). For a batch, the status of its last report.
cost_usd	What this post added, in US dollars. For a batch, the sum.
session_cost_usd	Everything recorded for the session so far.
budget.exceeded	true when a blocking budget on the key is at or past its limit, counting this session. budget.scope and budget.budget_id name it.
budget.unknown	Present and true when the budgets could not be read. exceeded is then false and means no verdict, not within budget.

The gateway cannot end a call. When budget.exceeded is true, close the socket from your client.

Closing a session

curl -X POST "https://gateway.langwatch.ai/v1/realtime/sessions/$SESSION_ID/close" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"duration_ms": 42000}'

The body is optional. A close records no usage: what the reports already recorded stands. Closing frees the key’s open-session slot.

Sessions that never report

A session past its open window with no close is settled by LangWatch:

• It sent reports. It is closed and the reported usage stands.
• It sent none. It is charged an estimate and closed. The assumed duration is the credential’s lifetime, or the default below when the lifetime is not known, held between 60 seconds and 60 minutes.

Session	Estimate	Default duration
OpenAI Realtime	10 input audio tokens per second, plus 20 output audio tokens per second for half the duration	600 s
ElevenLabs speech socket	15 characters per second	300 s
ElevenLabs transcription socket or batch	Audio for the whole duration	300 s

An ElevenLabs Conversational AI session is not estimated. It settles from the vendor’s own report, or as cost unknown.

OpenAI Live

POST /v1/live/sessions creates an OpenAI Live session over WebRTC. Send the body you would send to OpenAI: the session, and a transport of type webrtc carrying your SDP offer. The gateway resolves session.model, creates the session with the provider key and returns OpenAI’s answer with its own status.

curl -X POST "$LANGWATCH_GATEWAY_URL/v1/live/sessions" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session": { "model": "openai/gpt-live-1", "audio": { "output": { "voice": "marin" } } },
    "transport": { "type": "webrtc", "sdp": "v=0..." }
  }'

The answer is OpenAI’s body with langwatch.session_id added. The same id is on the X-LangWatch-Session-Id header. Set transport.sdp from the answer as the remote description. Audio then runs from the browser to OpenAI and does not pass through the gateway.

The gateway meters this session itself. It attaches to the session’s event socket at OpenAI with the provider key and records usage while the call runs:

• Duration. OpenAI states the session’s cumulative seconds. The gateway reports the difference since its last report, at most every 10 seconds and once more when the session closes.
• Delegated responses. A session that sets session.delegation.responses.model runs those responses on another model. Each completed response is recorded once, under that model’s token prices.

Your client sends no usage report and does not close the session.

Only the webrtc transport is served; any other transport.type answers 400 bad_request. The model in session.delegation.responses.model has to pass the key’s allowed models and policy rules, or the request answers 400 model_not_allowed.

WebRTC calls with gateway-side metering

POST /v1/realtime/calls creates an OpenAI Realtime call the same way. Use it in place of a client secret when the gateway should meter the call and be able to end it. The route takes both forms OpenAI’s own route takes.

Multipart, with the offer and the session declaration:

curl -X POST "$LANGWATCH_GATEWAY_URL/v1/realtime/calls" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY" \
  -F "sdp=<offer.sdp;type=application/sdp" \
  -F 'session={"type":"realtime","model":"openai/gpt-realtime-2.1"};type=application/json'

Raw SDP, with the model in the query string:

curl -X POST "$LANGWATCH_GATEWAY_URL/v1/realtime/calls?model=openai/gpt-realtime-2.1" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY" \
  -H "Content-Type: application/sdp" \
  --data-binary @offer.sdp

The response is OpenAI’s SDP answer, unchanged, with OpenAI’s Location header (/v1/realtime/calls/{call_id}) and X-LangWatch-Session-Id.

The gateway records every response.done and every input transcription the call produces, each once. Do not post these to /v1/realtime/sessions/{session_id}/usage as well. The session closes when OpenAI ends the call.

When the gateway ends a call

The gateway ends a call it created on either route, without the client, when:

Reason	Trigger
budget_exceeded	A budget that applies to the key is over its limit. Checked on every usage report and every 30 seconds.
key_revoked	The virtual key was revoked, deleted or expired. Noticed within 60 seconds.
sideband_lost	The gateway lost its event socket and could not attach again within 2 minutes, so it can no longer meter.
drain	The gateway instance is shutting down and the call outlived its drain time.
session_closed	LangWatch already closed the session, for example because it passed the open session window.

A Live session is asked to close on its socket, then hung up if OpenAI does not confirm. A Realtime call is hung up on /v1/realtime/calls/{call_id}/hangup. The usage recorded up to that point stands. A Live session whose event socket never attached is charged 15 seconds.

A Live client receives session.closed with reason close_requested on its data channel. A Realtime client sees its data channel close. Neither is told why the gateway ended the call; the reason is on the gateway_voice_sessions_ended_total metric and in the gateway log (voice_session_ended).

Each gateway instance supervises a bounded number of calls (LW_GATEWAY_VOICE_MAX_SUPERVISED_SESSIONS, default 2000). A request to an instance that is full or shutting down answers 503 voice_broker_unavailable with Retry-After, before anything is booked. On shutdown an instance keeps metering its calls for LW_GATEWAY_VOICE_DRAIN_SECONDS (default 570; the Helm chart sets shutdown.voiceDrainSeconds), then ends the ones still open.

WebSocket relay

OpenAI’s WebSocket transport needs the real API key on the socket, so a client on that transport cannot dial OpenAI with a virtual key. For it the gateway relays: the client opens the socket on the gateway with its virtual key, the gateway opens the vendor’s socket with the provider key and passes frames both ways.

WebRTC through /v1/live/sessions and /v1/realtime/calls stays the default, because media then runs client to OpenAI. The relay adds one hop to every frame. Measured on loopback, that hop costs about 11 microseconds per frame at the median and 20 at the 95th percentile, before network distance to the gateway.

The relay meters the session itself, like a brokered call. Your client sends no usage report.

OpenAI Realtime

GET /v1/realtime?model=... with a WebSocket upgrade. The model is resolved through the key’s aliases and allowlist before the upgrade, so a refused request answers over HTTP.

wscat -c "wss://gateway.langwatch.ai/v1/realtime?model=openai/gpt-realtime-2.1" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY"
> {"type":"response.create","response":{"output_modalities":["text"],"instructions":"Say hello."}}

A browser cannot set headers on a WebSocket. It passes the virtual key as a subprotocol, the way OpenAI’s own browser clients pass a key:

const ws = new WebSocket(
  "wss://gateway.langwatch.ai/v1/realtime?model=openai/gpt-realtime-2.1",
  ["realtime", `openai-insecure-api-key.${virtualKey}`],
);

The gateway answers the realtime subprotocol and never the key one. The key subprotocol, and any openai-organization. or openai-project. subprotocol, is not sent to OpenAI.

Every response.done and every input transcription is recorded once, as on a brokered call. The route is served by an OpenAI credential. A key with only an Azure OpenAI credential answers 400 model_provider_not_bound.

OpenAI Live

GET /v1/live/sessions with a WebSocket upgrade. OpenAI takes the model in the first frame, so the client’s first frame must be session.start:

wscat -c "wss://gateway.langwatch.ai/v1/live/sessions" \
  -H "Authorization: Bearer $LANGWATCH_VIRTUAL_KEY"
> {"type":"session.start","session":{"model":"openai/gpt-live-1","audio":{"format":{"type":"audio/pcm","rate":24000}}}}

A browser passes the virtual key as a subprotocol, as on the Realtime socket. The gateway answers realtime to the browser and opens OpenAI’s socket with no subprotocol:

const ws = new WebSocket("wss://gateway.langwatch.ai/v1/live/sessions", ["realtime", `openai-insecure-api-key.${virtualKey}`]);

The gateway reads that one frame, resolves session.model, checks session.delegation.responses.model against the allowlist, writes the resolved model back and sends the frame to OpenAI. A first frame that is not session.start, names no model, or names a model the key does not allow gets an error frame and close 1008. The client has 15 seconds to send it.

Duration is reported from OpenAI’s cumulative seconds and delegated responses under their own model, as on the WebRTC route. When the client leaves without session.close, the gateway sends it, so the final duration is the one OpenAI states.

What the relay reads

Frames	What the gateway does
Client to vendor	Passed on with the same type and the same bytes. Not parsed. The one exception is the session.start frame of a Live socket.
Vendor to client	Passed on with the same type and the same bytes. Text frames are read for the event type, and usage events (response.done, input transcription completed, session.usage.updated, session.closed, a delegated response.completed) for their ids and usage numbers.

Audio and transcripts are not logged or stored. A message can be up to 16 MiB.

When the relay closes the socket

A close from OpenAI reaches the client with OpenAI’s code and reason, and a close from the client reaches OpenAI the same way. The gateway closes the socket itself in these cases:

Close code	Reason	Meaning	What the client should do
1008	budget_exceeded	A budget that applies to the key is over its limit. Checked after every usage report and every 30 seconds.	Stop. A new socket answers 402 until the budget has room.
1008	key_revoked	The virtual key was revoked, disabled, deleted or expired. Noticed within 60 seconds.	Stop. The key no longer works.
1008	session_closed	LangWatch already closed the session.	Open a new socket.
1012	service_restart	The gateway instance is being replaced by a deploy.	Open a new socket at once. Another instance answers.
1011	provider_connection_lost	The vendor socket dropped without a close frame.	Open a new socket.

Before a budget close the client receives one text frame:

{"type":"error","error":{"type":"budget_exceeded","code":"budget_exceeded","message":"The budget on this key is exhausted, so the gateway is closing this session."}}

On a deploy the instance stops taking upgrades and answers 503 voice_broker_unavailable with Retry-After. Open sockets keep running for LW_GATEWAY_VOICE_DRAIN_SECONDS, then close with 1012. A voice session does not carry over to another instance, so the client starts a new one and resends its session configuration.

Relayed sockets count toward LW_GATEWAY_VOICE_MAX_SUPERVISED_SESSIONS and toward the key’s max open sessions.

Limiting how many calls a key runs at once

Set max open sessions on the virtual key. The request-rate limits do not bound voice: one mint is a single request that opens a call billing for as long as it runs, so a key at 60 requests per minute can hold sixty ten-minute calls without tripping anything.

A mint over the cap answers 429 realtime_session_limit, and the refusal is recorded in the spend surface. A slot frees when the session closes. A session older than the longest call a vendor allows stops holding a slot, which matters for OpenAI, whose socket never signals that it closed.

What the broker does not do

• It does not proxy the voice socket of a minted or WebRTC session. Only the WebSocket relay carries frames, and it records usage, not content. There is no frame-level tracing. A session minted as a client secret, signed URL or single-use token cannot be ended in progress; a call created on /v1/live/sessions or /v1/realtime/calls can.
• It does not enforce the session’s tools. A hosted agent’s tools live at the vendor, and an OpenAI session declares its own.
• Guardrails do not run on a mint. The body is a session declaration, not a prompt, and the conversation never reaches the gateway. The response says so on X-LangWatch-Guardrails-Not-Applied: realtime_session instead of reporting a check that could not mean anything.

Turn-level transcripts and latency reach LangWatch through the SDK in your own application.

Errors

Code	Status	Cause
bad_request	400	agent_id is missing on the ElevenLabs mint, the token type is unknown, or the usage body is not one of the accepted shapes.
not_found	404	No session with that id belongs to this key.
model_provider_not_bound	400	The key has no credential for the vendor this route serves.
budget_exceeded	402	A budget that applies to the key is over its limit.
realtime_session_limit	429	The key holds its maximum of open voice sessions. Retry when a call ends.
realtime_registry_unavailable	503	LangWatch could not record the session, so no session was minted. Retry.
model_not_allowed	400	The session model, or the model a Live session delegates responses to, is outside the key’s allowed models.
voice_broker_unavailable	503	The gateway instance is full or shutting down. No call was created. Retry after Retry-After.

Each mint route serves one vendor: /v1/realtime/client_secrets needs an OpenAI credential, and /v1/convai/conversation/get-signed-url and /v1/single-use-token/{token_type} need an ElevenLabs one. A mint never falls back to a second credential, because a session credential is bound to one account and a fallback would sign for an agent that does not exist there.

Also check: ElevenLabs, Budgets, Errors.