> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Helm chart

> Deploy the LangWatch AI Gateway alongside your self-hosted LangWatch.

The gateway is a sub-chart of the `langwatch` Helm chart. The umbrella chart enables it by default with `gateway.chartManaged: true` and runs it as its own Deployment and Service next to `langwatch-app`. The gateway gets no Ingress of its own: publish it with `ingress.gateway.host` on the app Ingress. This guide takes you from an empty namespace to a gateway that answers a chat completion.

## Prerequisites

* A Kubernetes cluster and `helm` 3.
* An ingress controller or a Gateway API implementation. [Routing and ingress controllers](https://langwatch.ai/docs/self-hosting/deployment/routing.md) lists what to expose on each.
* A hostname for the gateway, separate from the app hostname. See [DNS and TLS](https://langwatch.ai/docs/ai-gateway/self-hosting/dns-and-tls.md).
* The LangWatch umbrella chart. Follow [Kubernetes (Helm)](https://langwatch.ai/docs/self-hosting/deployment/kubernetes-helm.md) for the app side first if you have not installed it yet.

**1. Create the app Secret**

The gateway pod and the app pod mount the same Secret. The gateway reads two keys from it: `LW_GATEWAY_INTERNAL_SECRET` signs every call between the gateway and the control plane, and `LW_GATEWAY_JWT_SECRET` verifies the JWTs the control plane issues for resolved virtual keys.

With `autogen.enabled=true` the chart creates the Secret on first install. Otherwise create it before `helm install`:

```bash theme={null}
kubectl create namespace langwatch

kubectl -n langwatch create secret generic langwatch-app-secrets \
  --from-literal=credentialsEncryptionKey="$(openssl rand -hex 32)" \
  --from-literal=nextAuthSecret="$(openssl rand -hex 32)" \
  --from-literal=virtualKeyPepper="$(openssl rand -hex 32)" \
  --from-literal=LW_GATEWAY_INTERNAL_SECRET="$(openssl rand -hex 32)" \
  --from-literal=LW_GATEWAY_JWT_SECRET="$(openssl rand -hex 32)"
```

`gateway.secrets.existingSecretName` (default `langwatch-app-secrets`) must equal the app Secret name. If you set `secrets.existingSecret` or `autogen.secretNames.app` to another name, set `gateway.secrets.existingSecretName` to the same name. The chart refuses to render when the two differ.

`virtualKeyPepper` is read by the app only. The gateway never mounts it.

**2. Write the gateway values**

Add a `gateway` block to your umbrella `values.yaml`, and the gateway host to the app `ingress` block:

```yaml theme={null}
ingress:
  enabled: true
  className: nginx
  # ...your app hosts, tls and annotations
  gateway:
    host: gateway.acme.com        # publishes /v1 and /health on this host
    tls:
      secretName: gateway-tls
  # className nginx adds the streaming, timeout and body-size annotations
  # to the gateway Ingress only.

gateway:
  chartManaged: true

  # Externally reachable URL the control plane hands to CLI users at login.
  # Leave empty to derive https://<ingress.gateway.host> from the ingress below.
  publicUrl: ""

  # Where the gateway reaches the control plane. Empty resolves to
  # http://<release>-app:5560, the app Service of this install.
  controlPlane:
    baseUrl: ""

  secrets:
    existingSecretName: langwatch-app-secrets
    internalSecretKey: LW_GATEWAY_INTERNAL_SECRET
    jwtSecretKey: LW_GATEWAY_JWT_SECRET
    jwtSecretPreviousKey: "" # set only during a JWT secret rotation

  security:
    maxRequestBodyBytes: 33554432 # 32 MiB, applied before auth
    blockLocalHTTPCalls: false
    requireHTTPSCustomEndpoints: false
    allowedProxyHosts: []
    extraCACertsConfigMap: "" # ConfigMap of PEM CA certificates to trust
    corsAllowedOrigins: [] # browser origins allowed to call /v1

  # OTLP/HTTP endpoint for the gateway's own spans. Empty disables export.
  otel:
    endpoint: ""

  spend:
    enabled: true # off means budgets are never debited

  shutdown:
    preDrainWaitSeconds: 5
    timeoutSeconds: 60
  terminationGracePeriodSeconds: 75
```

The image is `docker.io/langwatch/ai-gateway`. `image.tag` defaults to the chart `appVersion`. Set `image.repository` to your mirror if the cluster cannot pull from Docker Hub.

**3. Install or upgrade**

```bash theme={null}
helm repo add langwatch https://langwatch.github.io/langwatch
helm repo update

helm upgrade --install langwatch langwatch/langwatch \
  --namespace langwatch --create-namespace \
  -f values.yaml \
  --wait --timeout 45m
```

The umbrella chart bundles the gateway sub-chart, so one `helm upgrade` rolls the app and the gateway together. The gateway Deployment uses a `RollingUpdate` with `maxSurge: 1` and `maxUnavailable: 0`.

**4. Verify**

```bash theme={null}
kubectl -n langwatch rollout status deploy/langwatch-gateway
kubectl -n langwatch run curl --rm -it --image=curlimages/curl --restart=Never \
  -- curl -sS http://langwatch-gateway/health
```

Expected output:

```json theme={null}
{ "status": "ok", "checks": { "gateway": "ok", "control_plane": "ok" } }
```

Then run the [post-install checklist](https://langwatch.ai/docs/ai-gateway/self-hosting/post-install-checklist.md) to send a first completion through the gateway.

## What the chart renders into the pod

The chart passes these environment variables to the container. Every other key in the gateway `values.yaml` (`cache.*`, `bifrost.*`, `guardrails.*`, `budget.*`, `startup.*`, `otel.auth*`, `otel.batchTimeout`, `otel.maxQueueSize`, `security.*Timeout`, `controlPlane.*Timeout`) does not reach the pod.

| Env var | Value source | Purpose |
| - | - | - |
| `SERVER_ADDR` | fixed `:5563` | Listen address. The Service targets port 5563 |
| `LOG_LEVEL` | `logging.level` | `debug`, `info`, `warn` or `error` |
| `LW_GATEWAY_BASE_URL` | `controlPlane.baseUrl`, else `http://<release>-app:5560` | Control plane URL |
| `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_OTLP_ENDPOINT` | `otel.endpoint`, only when set | OTLP/HTTP endpoint for the gateway's own spans |
| `LW_GATEWAY_SPEND_ENABLED` | `spend.enabled` | Spend record emission |
| `OTEL_METRICS_EXPORTER`, `OTEL_EXPORTER_PROMETHEUS_PORT` | `otlp,prometheus` and `metrics.port`, only when `metrics.enabled` is `true` | Metrics listener, default port 9464 |
| `METRICS_API_KEY` | Secret key `metrics.apiKey.secretKey` (default `METRICS_API_KEY`), in `metrics.apiKey.existingSecretName` or else `secrets.existingSecretName`, only when `metrics.enabled` is `true` | Bearer a scrape sends |
| `SERVER_MAX_REQUEST_BODY_BYTES` | `security.maxRequestBodyBytes` | Request body cap |
| `SERVER_DRAIN_DELAY_SECONDS` | `shutdown.preDrainWaitSeconds` | Wait after `/readyz` flips to 503 |
| `SERVER_GRACEFUL_SECONDS` | `shutdown.timeoutSeconds` | In-flight request budget on shutdown |
| `BLOCK_LOCAL_HTTP_CALLS` | `security.blockLocalHTTPCalls` | Refuse provider endpoints on private addresses |
| `REQUIRE_HTTPS_CUSTOM_ENDPOINTS` | `security.requireHTTPSCustomEndpoints` | Refuse `http://` provider endpoints |
| `ALLOWED_PROXY_HOSTS` | `security.allowedProxyHosts`, comma-joined | Private hostnames exempt from the block |
| `SSL_CERT_DIR` | set to `/etc/langwatch/ca` when `security.extraCACertsConfigMap` is set | Extra CA certificates to trust |
| `LW_GATEWAY_CORS_ALLOWED_ORIGINS` | `security.corsAllowedOrigins`, comma-joined | Browser origins allowed to call `/v1` |
| `LW_GATEWAY_INTERNAL_SECRET` | Secret `secrets.existingSecretName`, key `secrets.internalSecretKey` | HMAC secret for control plane calls |
| `LW_GATEWAY_JWT_SECRET` | Secret `secrets.existingSecretName`, key `secrets.jwtSecretKey` | JWT verification secret |
| `LW_GATEWAY_JWT_SECRET_PREVIOUS` | Secret, key `secrets.jwtSecretPreviousKey`, only when set | Previous JWT secret during rotation |

The chart renders no authentication header for the OTLP export. Point `otel.endpoint` at a collector that accepts unauthenticated OTLP/HTTP, or leave it empty. The full variable reference is on [Config](https://langwatch.ai/docs/ai-gateway/self-hosting/config.md).

Write `maxRequestBodyBytes` as an integer literal. The chart renders it with `%d`, and a YAML float such as `3.3554432e+07` fails the render.

## Rotate the JWT secret without downtime

The gateway verifies a JWT against `LW_GATEWAY_JWT_SECRET` first, then against `LW_GATEWAY_JWT_SECRET_PREVIOUS` when that variable is set. The control plane signs with the `LW_GATEWAY_JWT_SECRET` key of the same Secret, and a JWT lives 15 minutes.

1. Edit the Secret: copy the current `LW_GATEWAY_JWT_SECRET` value into a new key `LW_GATEWAY_JWT_SECRET_PREVIOUS`, then replace `LW_GATEWAY_JWT_SECRET` with the new value.
2. Set `gateway.secrets.jwtSecretPreviousKey: LW_GATEWAY_JWT_SECRET_PREVIOUS` and run `helm upgrade`. The gateway pods restart with both values. Wait for `kubectl -n langwatch rollout status deploy/langwatch-gateway`.
3. Run `kubectl -n langwatch rollout restart deploy/langwatch-app`. New JWTs are signed with the new value; JWTs signed before the restart still verify against the previous value.
4. After 15 minutes, delete the `LW_GATEWAY_JWT_SECRET_PREVIOUS` key from the Secret, clear `jwtSecretPreviousKey`, and run `helm upgrade` again.

`LW_GATEWAY_INTERNAL_SECRET` has no previous-key slot. Change it in the Secret and restart the app and the gateway Deployments together.

## Control plane dependency

The gateway calls these routes on `LW_GATEWAY_BASE_URL`, signed with `LW_GATEWAY_INTERNAL_SECRET`: `/api/internal/gateway/resolve-key`, `/config`, `/changes`, `/health`, `/budget-bucket-spend` and `/realtime-sessions`. It also ships spend records to the control plane's spend ingest.

When the control plane is unreachable, virtual keys already in the auth cache keep serving under the stale-while-error rules on [Config](https://langwatch.ai/docs/ai-gateway/self-hosting/config.md#auth-cache). Keys not in the cache fail with `503 auth_upstream_unavailable`.

Budget debits come from the spend records the gateway ships. With `spend.enabled: false` the gateway serves traffic and debits no budget. The spool lives in the pod's `/tmp` emptyDir (`scratchSizeLimit`, default `1Gi`) and is capped at 64 MiB. Alert on `gateway_spend_spool_dropped_total`: every drop is spend that is never debited.

## Graceful drain

On `SIGTERM` the gateway:

1. Flips `/readyz` to `503 {"status":"draining"}` and sets the `gateway_draining` gauge to 1.
2. Waits `shutdown.preDrainWaitSeconds` (default 5) so the endpoint controller removes the pod from the Service.
3. Stops the HTTP listener and gives in-flight requests up to `shutdown.timeoutSeconds` (default 60) to finish, then stops the other services in reverse start order. The spend spool closes after the listener, so requests that finish during the drain still record spend.

`terminationGracePeriodSeconds` must be at least `preDrainWaitSeconds + timeoutSeconds + 10`. The chart refuses to render otherwise. It also refuses the retired keys `shutdown.preDrainWait` and `shutdown.timeout`.

A streaming response can run up to 14 minutes, the provider request timeout. With the default `timeoutSeconds: 60` the gateway logs `graceful_shutdown_shorter_than_max_stream_duration` at boot, and a stream still open after 60 seconds of drain is cut. To finish long streams on every rollout:

```yaml theme={null}
gateway:
  shutdown:
    preDrainWaitSeconds: 5
    timeoutSeconds: 840
  terminationGracePeriodSeconds: 855
```

Watch `gateway_draining` together with `gateway_in_flight_requests`. During a healthy drain the in-flight count falls to 0. A flat count past the window is a stuck handler, and the kubelet kills the pod when the grace period ends.

## NetworkPolicy

`networkPolicy.enabled` is `false` by default. When enabled the chart renders one deny-by-default policy:

| Direction | Allowed | Value |
| - | - | - |
| Ingress | Pods in the `ingress-nginx` namespace, to port 5563 | `networkPolicy.ingressFrom` |
| Ingress | Prometheus pods in the `monitoring` namespace, to the metrics port, only when `metrics.enabled` is `true` | `networkPolicy.metricsFrom` |
| Egress | DNS to `kube-system` on TCP and UDP 53 | fixed |
| Egress | Pods labelled `app.kubernetes.io/name: langwatch-app` on port 5560 | `networkPolicy.egressToControlPlane`, `networkPolicy.controlPlanePort` |
| Egress | OTLP collector on port 4318, only when the list is non-empty | `networkPolicy.egressToOTLP` |
| Egress | Any public IP on TCP 443, with private, loopback, link-local and cloud metadata ranges excluded | `networkPolicy.egressToProviders`, `networkPolicy.providerPorts` |

Replace `egressToProviders` with explicit CIDR blocks when your policy requires a pinned provider egress set. Check the namespace labels on your cluster before enabling: the defaults match upstream ingress-nginx and kube-prometheus-stack.

Confirm the policy renders:

```bash theme={null}
helm template langwatch langwatch/langwatch -f values.yaml \
  --set gateway.networkPolicy.enabled=true | grep -A2 'kind: NetworkPolicy'
```

## Pod security

The pod runs as user and group `65532` with `runAsNonRoot`, a read-only root filesystem, all capabilities dropped, the `RuntimeDefault` seccomp profile and `automountServiceAccountToken: false`. The only writable path is the `/tmp` emptyDir. Adjust `podSecurityContext` and `containerSecurityContext` only if an admission policy on your cluster requires it.

## Roll back

```bash theme={null}
helm history langwatch -n langwatch
helm rollback langwatch <revision> -n langwatch
```

Every new gateway pod starts with an empty auth cache and resolves each virtual key from the control plane on its first request.

## Common failures

| Symptom | Cause | Fix |
| - | - | - |
| Pod in `CreateContainerConfigError` | The Secret named by `gateway.secrets.existingSecretName` or one of its keys does not exist | Create the Secret from step 1, then delete the pod |
| `helm upgrade` fails naming `gateway.secrets.existingSecretName` | The gateway Secret name differs from the app Secret name | Set both to the same name |
| `helm upgrade` fails naming `terminationGracePeriodSeconds` | Drain timing exceeds the grace period | Raise `terminationGracePeriodSeconds` to the number in the message |
| Pods healthy, every `/v1` call answers `503 auth_upstream_unavailable` | `LW_GATEWAY_BASE_URL` points at the wrong host or port | Set `gateway.controlPlane.baseUrl` to the app Service URL |
| `/health` answers 503 with `control_plane: unreachable for Ns` | Control plane unreachable, or `LW_GATEWAY_INTERNAL_SECRET` differs between the two pods | Compare the value in both Deployments, then check the Service and DNS |
| Pod refuses to start with `LW_GATEWAY_AUTH_CACHE_... is no longer read` | A retired duration-string variable is still set | Set the `_SECONDS` variable named in the message instead |

More symptoms are on [Troubleshooting](https://langwatch.ai/docs/ai-gateway/troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.