> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Grafana dashboard

> An importable Grafana dashboard for the LangWatch AI Gateway, covering request health, provider health, the auth cache, budgets, guardrails, streaming and the drain.

The JSON below builds nine rows that an on-call engineer reads top down, from caller impact to the cause. Every panel reads a metric the gateway registers, so it works against any Prometheus that scrapes the gateway's `/metrics` on the metrics port.

Pair it with [Prometheus alerts](https://langwatch.ai/docs/ai-gateway/cookbooks/prometheus-alerts.md) for the paging rules, and [Troubleshooting](https://langwatch.ai/docs/ai-gateway/troubleshooting.md) for the fix behind each row.

## What it covers

Nine rows, top to bottom:

| Row | Panels | What it answers |
| - | - | - |
| Request health | rate, p50, p95, p99, 5xx share | Is this affecting callers? |
| Provider health | circuit state, upstream latency, fallback events | Which credential is degrading, and is fallback covering it? |
| Auth cache | hit rate, resolve-key round trips | Is cache pressure loading the control plane? |
| Budgets | blocks by scope, spend records written and dropped | Which scope is rejecting traffic, and is spend reaching the ledger? |
| Guardrails | verdicts by direction, blocks by direction | Policy enforcement against a policy regression |
| Streaming | active streams, guardrail fail-open rate | Concurrency, and chunks passing through unchecked |
| Prompt cache | hit and miss rate | Whether provider cache reads are happening at all |
| Infrastructure | replicas, CPU, memory, goroutines | The Kubernetes side of everything above |
| Lifecycle | draining pods, in-flight requests | A stuck drain, before the kubelet kills the pod |

## Prerequisites

* Grafana 10.0 or later. The JSON uses time series panels and `$__rate_interval`.
* A Prometheus data source that scrapes the gateway's metrics port (`9464`) with the bearer `METRICS_API_KEY`.
* `kube-state-metrics` and cAdvisor for the infrastructure row. Skip that row without them.

## Import

Save the JSON below as `langwatch-gateway.json`, then in Grafana open **Dashboards → New → Import → Upload JSON**. Set the data source on first import; it reaches every panel.

```json theme={null}
{
  "title": "LangWatch AI Gateway",
  "tags": ["langwatch", "ai-gateway", "llm"],
  "timezone": "browser",
  "schemaVersion": 39,
  "version": 1,
  "refresh": "30s",
  "time": { "from": "now-1h", "to": "now" },
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus",
        "current": { "text": "Prometheus", "value": "Prometheus" }
      },
      {
        "name": "namespace",
        "type": "query",
        "datasource": "$datasource",
        "query": "label_values(gateway_http_requests_total, namespace)",
        "current": { "text": "langwatch", "value": "langwatch" },
        "includeAll": false
      },
      {
        "name": "pod",
        "type": "query",
        "datasource": "$datasource",
        "query": "label_values(gateway_http_requests_total{namespace=\"$namespace\"}, pod)",
        "includeAll": true,
        "multi": true
      }
    ]
  },
  "panels": [
    {
      "type": "row",
      "title": "Request health",
      "gridPos": { "x": 0, "y": 0, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Request rate",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 1, "w": 8, "h": 7 },
      "targets": [
        {
          "expr": "sum by (status) (rate(gateway_http_requests_total{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))",
          "legendFormat": "{{status}}"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "reqps" } }
    },
    {
      "type": "timeseries",
      "title": "Latency (p50 / p95 / p99)",
      "datasource": "$datasource",
      "gridPos": { "x": 8, "y": 1, "w": 8, "h": 7 },
      "targets": [
        {
          "expr": "histogram_quantile(0.50, sum by (le) (rate(gateway_http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
          "legendFormat": "p50"
        },
        {
          "expr": "histogram_quantile(0.95, sum by (le) (rate(gateway_http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
          "legendFormat": "p95"
        },
        {
          "expr": "histogram_quantile(0.99, sum by (le) (rate(gateway_http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))",
          "legendFormat": "p99"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "s" } }
    },
    {
      "type": "stat",
      "title": "5xx rate",
      "datasource": "$datasource",
      "gridPos": { "x": 16, "y": 1, "w": 8, "h": 7 },
      "targets": [
        {
          "expr": "sum(rate(gateway_http_requests_total{namespace=\"$namespace\", status=~\"5..\"}[$__rate_interval])) / sum(rate(gateway_http_requests_total{namespace=\"$namespace\"}[$__rate_interval]))"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 0.01 },
              { "color": "red", "value": 0.05 }
            ]
          }
        }
      }
    },

    {
      "type": "row",
      "title": "Provider health",
      "gridPos": { "x": 0, "y": 8, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Circuit state by credential (0=closed 1=open 2=half-open)",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 9, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "gateway_circuit_state{namespace=\"$namespace\", pod=~\"$pod\"}",
          "legendFormat": "{{credential_id}}"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Upstream latency p95 by provider",
      "datasource": "$datasource",
      "gridPos": { "x": 12, "y": 9, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum by (provider, le) (rate(gateway_provider_duration_seconds_bucket{namespace=\"$namespace\"}[$__rate_interval])))",
          "legendFormat": "{{provider}}"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "s" } }
    },
    {
      "type": "timeseries",
      "title": "Fallback events per sec",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 16, "w": 24, "h": 6 },
      "targets": [
        {
          "expr": "sum by (from_credential, to_credential) (rate(gateway_fallback_events_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "{{from_credential}} → {{to_credential}}"
        }
      ]
    },

    {
      "type": "row",
      "title": "Auth cache",
      "gridPos": { "x": 0, "y": 22, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Auth cache hit rate",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 23, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum(rate(gateway_auth_cache_hits_total{namespace=\"$namespace\", tier=\"l1\"}[$__rate_interval])) / sum(rate(gateway_auth_cache_lookups_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "hit rate"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "percentunit" } }
    },
    {
      "type": "timeseries",
      "title": "Resolve-key round-trips to control plane",
      "datasource": "$datasource",
      "gridPos": { "x": 12, "y": 23, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum(rate(gateway_control_plane_requests_total{namespace=\"$namespace\", endpoint=\"resolve-key\"}[$__rate_interval]))"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "reqps" } }
    },

    {
      "type": "row",
      "title": "Budgets",
      "gridPos": { "x": 0, "y": 30, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Budget blocks by scope",
      "description": "The gateway records spend and the control plane writes the ledger. This counter is the only gateway-side signal that a hard cap rejected traffic.",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 31, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum by (scope) (rate(gateway_budget_blocks_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "{{scope}}"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Spend records written and dropped",
      "description": "Budget debits come from these records. A drop is spend that is never billed and never enforced against.",
      "datasource": "$datasource",
      "gridPos": { "x": 12, "y": 31, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum(rate(gateway_spend_spool_appended_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "written"
        },
        {
          "expr": "sum by (reason) (rate(gateway_spend_spool_dropped_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "dropped / {{reason}}"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "reqps" } }
    },

    {
      "type": "row",
      "title": "Guardrails",
      "gridPos": { "x": 0, "y": 38, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Verdicts by direction",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 39, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum by (direction, verdict) (rate(gateway_guardrail_verdicts_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "{{direction}} / {{verdict}}"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Blocks by direction",
      "datasource": "$datasource",
      "gridPos": { "x": 12, "y": 39, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum by (direction) (rate(gateway_guardrail_verdicts_total{namespace=\"$namespace\", verdict=\"block\"}[$__rate_interval]))",
          "legendFormat": "{{direction}}"
        }
      ]
    },

    {
      "type": "row",
      "title": "Streaming",
      "gridPos": { "x": 0, "y": 46, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Active streams",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 47, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum(gateway_streaming_active{namespace=\"$namespace\", pod=~\"$pod\"})"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Guardrail fail-open rate",
      "description": "Emitted on any direction when the control-plane guardrail call fails and the gateway lets the traffic through unchecked. Sustained > 0 means the evaluator is missing its SLO. The stream_chunk series is expected to be the noisy one, because that direction always fails open so a slow policy service never stalls a stream; treat it as the chunk-level enforcement gap. Use the langwatch.guardrail.stream_chunk_fail_open trace attribute to see why an individual bypass happened.",
      "datasource": "$datasource",
      "gridPos": { "x": 12, "y": 47, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum by (direction) (rate(gateway_guardrail_verdicts_total{namespace=\"$namespace\", verdict=\"fail_open\"}[$__rate_interval]))",
          "legendFormat": "{{direction}}"
        }
      ]
    },

    {
      "type": "row",
      "title": "Prompt cache",
      "gridPos": { "x": 0, "y": 54, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Prompt cache outcomes",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 55, "w": 24, "h": 7 },
      "targets": [
        {
          "expr": "sum by (outcome) (rate(gateway_cache_hits_total{namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "{{outcome}}"
        }
      ]
    },

    {
      "type": "row",
      "title": "Infrastructure",
      "gridPos": { "x": 0, "y": 62, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Pod replicas ready",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 63, "w": 8, "h": 7 },
      "targets": [
        {
          "expr": "sum(kube_deployment_status_replicas_available{namespace=\"$namespace\", deployment=\"langwatch-gateway\"})"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "CPU + memory per pod",
      "datasource": "$datasource",
      "gridPos": { "x": 8, "y": 63, "w": 8, "h": 7 },
      "targets": [
        {
          "expr": "sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", pod=~\"langwatch-gateway-.*\"}[$__rate_interval]))",
          "legendFormat": "cpu / {{pod}}"
        },
        {
          "expr": "sum by (pod) (container_memory_working_set_bytes{namespace=\"$namespace\", pod=~\"langwatch-gateway-.*\"}) / 1024 / 1024",
          "legendFormat": "mem MB / {{pod}}"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Goroutines per pod",
      "description": "Monotonic climb = goroutine leak. See the AI Gateway troubleshooting page.",
      "datasource": "$datasource",
      "gridPos": { "x": 16, "y": 63, "w": 8, "h": 7 },
      "targets": [
        {
          "expr": "go_goroutines{namespace=\"$namespace\", pod=~\"langwatch-gateway-.*\"}",
          "legendFormat": "{{pod}}"
        }
      ]
    },

    {
      "type": "row",
      "title": "Lifecycle (drain)",
      "gridPos": { "x": 0, "y": 70, "w": 24, "h": 1 }
    },
    {
      "type": "timeseries",
      "title": "Draining pods",
      "description": "gateway_draining flips to 1 on SIGTERM; a pod stuck at 1 for > terminationGracePeriodSeconds is a hung handler (see the AI Gateway troubleshooting page).",
      "datasource": "$datasource",
      "gridPos": { "x": 0, "y": 71, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum by (pod) (gateway_draining{namespace=\"$namespace\"})",
          "legendFormat": "{{pod}}"
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "In-flight requests",
      "description": "Paired with Draining. A healthy drain curves to 0; flat while draining=1 means a handler is hung.",
      "datasource": "$datasource",
      "gridPos": { "x": 12, "y": 71, "w": 12, "h": 7 },
      "targets": [
        {
          "expr": "sum by (pod) (gateway_in_flight_requests{namespace=\"$namespace\"})",
          "legendFormat": "{{pod}}"
        }
      ]
    }
  ]
}
```

## Read it top down

1. **Request health.** Latency and the 5xx share together answer whether callers are affected.
2. **Provider health.** An open circuit means the gateway is skipping a credential. Fallback events next to it say whether another credential is covering the traffic.
3. **Auth cache.** Latency up with providers healthy points at cache misses and the control-plane round trips they cost.
4. **Budgets.** Blocks climbing on a scope means a hard cap is rejecting traffic. Spend records dropped means budgets are being enforced against incomplete spend.
5. **Guardrails.** `direction` and `verdict` separate policy working from policy going too wide.
6. **Streaming.** Active streams is concurrency. The fail-open rate is chunks passing through unchecked, which is by design on the stream-chunk direction.
7. **Prompt cache.** A prefix-heavy workload with near-zero hits means the provider is not reading its cache.
8. **Infrastructure.** Goroutines climbing and never falling is a leak. Replicas and CPU connect autoscaler pressure to everything above.
9. **Lifecycle.** A pod with `gateway_draining` at 1 and a flat in-flight count is a stuck handler that the kubelet will kill.

## Link it from your alerts

Put the dashboard URL in each alert's `runbook_url`, or in the notification body:

```yaml theme={null}
receivers:
  - name: slack-gateway
    slack_configs:
      - channel: "#ai-gateway-alerts"
        title: "{{ .CommonAnnotations.summary }}"
        text: |
          {{ .CommonAnnotations.description }}

          <https://grafana.internal/d/langwatch-ai-gateway|Dashboard>
```

The engineer lands on the dashboard, narrows the cause to one row, then opens the matching section of [Troubleshooting](https://langwatch.ai/docs/ai-gateway/troubleshooting.md).

## See also

* [Prometheus alerts](https://langwatch.ai/docs/ai-gateway/cookbooks/prometheus-alerts.md): the rules that say when to open this dashboard.
* [Troubleshooting](https://langwatch.ai/docs/ai-gateway/troubleshooting.md): what to do once a row points at the cause.
* [Observability](https://langwatch.ai/docs/ai-gateway/observability.md): every metric, its labels, and the span attributes next to them.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.