> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Observability & Monitoring

> Monitor LangWatch infrastructure with Prometheus, Grafana, and health checks

LangWatch exposes Prometheus metrics and health check endpoints for monitoring your self-hosted deployment.

## Prometheus

The Helm chart includes an optional Prometheus instance that scrapes metrics from LangWatch components.

### Enable Prometheus

```yaml theme={null}
# In your values.yaml
app:
  telemetry:
    metrics:
      enabled: true
      apiKey:
        value: "your-metrics-api-key" # Authenticates scrape requests

prometheus:
  chartManaged: true
  server:
    retention: 30d
    persistentVolume:
      size: 20Gi
```

### What Gets Scraped

| Component | Port | Endpoint | Metrics |
| - | - | - | - |
| App | 5560 | `/metrics` | Trace ingestion, ClickHouse query latency, event loop lag |
| Workers | 2999 | `/metrics` | Queue backlog, job processing time, job success/failure rates |

Each component has its own scrape job. A job keeps only pods annotated `prometheus.io/scrape: "true"` whose `app.kubernetes.io/name` label is that component's (`<release>-app` or `<release>-workers`). The chart adds the annotation to the app and workers pods when `app.telemetry.metrics.enabled` is `true`. Neither component serves `/metrics` until `app.telemetry.metrics.enabled` is on with an API key, and then both require that key as a bearer token.

The chart-managed Prometheus sends the key only when it is set through `app.telemetry.metrics.apiKey.value`. With `apiKey.secretKeyRef`, the pods get the key but the scrape config does not, and both jobs are answered with 401.

### Access Prometheus

Port-forward to the Prometheus UI:

```bash theme={null}
kubectl -n langwatch port-forward svc/langwatch-prometheus-server 9090:9090
# Open http://localhost:9090
```

### External Prometheus

To use an existing Prometheus instance instead of the chart-managed one:

```yaml theme={null}
prometheus:
  chartManaged: false
  external:
    existingSecret: prometheus-credentials
    secretKeys:
      host: "host"
      port: "port"
      username: "username"
      password: "password"
```

You'll need to configure your external Prometheus to scrape the LangWatch pods, sending the metrics API key as a bearer token. With `app.telemetry.metrics.enabled: true`, the app pods are annotated with:

```yaml theme={null}
prometheus.io/scrape: "true"
prometheus.io/port: "5560"
```

The workers pods are annotated with:

```yaml theme={null}
prometheus.io/scrape: "true"
prometheus.io/port: "2999"
```

Both serve the metrics on the path `/metrics`.

## Grafana

Connect Grafana to your Prometheus instance to visualize LangWatch metrics.

### Key Dashboards

Set up dashboards for:

* **Span throughput**: spans ingested per minute
* **Worker queue backlog**: groups waiting to be processed and the age of the oldest pending job (indicates processing bottleneck)
* **ClickHouse query latency**: p50/p95/p99 query times
* **Resource utilization**: CPU and memory per component

### Example Queries

```promql theme={null}
# Span ingestion rate (per minute), by outcome
sum by (outcome) (rate(trace_ingestion_spans_total[5m])) * 60

# Worker queue backlog
sum by (queue_name) (gq_pending_groups)

# Age of the oldest due job, in seconds, including groups held in retry backoff
max by (queue_name) (gq_oldest_backlog_age_milliseconds) / 1000

# ClickHouse query p95 latency
histogram_quantile(0.95, sum by (le) (rate(clickhouse_query_duration_seconds_bucket[5m])))
```

## Health & Readiness Checks

### Endpoints

| Component | Endpoint | Method | Healthy Response |
| - | - | - | - |
| App | `/api/health` | GET | 200 OK |
| Workers | `/healthz` | GET | 200 OK |

### Kubernetes Probes

The Helm chart configures probes automatically. The app probes `/api/health` on port 5560:

```yaml theme={null}
# Startup probe (allows up to 40 minutes: on a first install the app runs the upgrade before it serves)
startupProbe:
  httpGet:
    path: /api/health
    port: http
  periodSeconds: 15
  failureThreshold: 160

# Readiness probe (removes from service if unhealthy)
readinessProbe:
  httpGet:
    path: /api/health
    port: http
  periodSeconds: 10
  timeoutSeconds: 5

# Liveness probe (restarts unhealthy pods)
livenessProbe:
  httpGet:
    path: /api/health
    port: http
  periodSeconds: 30
  timeoutSeconds: 5
```

The workers use the same startup budget and a liveness probe every 30 seconds, both on `/healthz` at port 2999. They have no readiness probe.

### Manual Health Check

```bash theme={null}
# Check app health
kubectl -n langwatch exec deploy/langwatch-app -- \
  curl -s http://localhost:5560/api/health

# Check worker health
kubectl -n langwatch exec deploy/langwatch-workers -- \
  curl -s http://localhost:2999/healthz
```

## Alerting Recommendations

Set up alerts for these critical conditions:

| Alert | Condition | Severity |
| - | - | - |
| Worker queue backlog | Oldest pending job older than 5 min | Warning |
| Worker queue backlog (critical) | Oldest pending job older than 30 min | Critical |
| ClickHouse memory | Memory usage > 80% of limit | Warning |
| ClickHouse disk | Hot storage > 85% full | Critical |
| PostgreSQL connections | Active connections > 80% of max | Warning |
| App error rate | HTTP 5xx rate > 5% for 5 min | Critical |
| Pod restarts | Pod restart count > 3 in 15 min | Warning |

### Example Alertmanager Rule

```yaml theme={null}
groups:
  - name: langwatch
    rules:
      - alert: WorkerQueueBacklog
        expr: max by (queue_name) (gq_oldest_backlog_age_milliseconds) > 300000
        labels:
          severity: warning
        annotations:
          summary: "Worker queue backlog is high"
          description: "Oldest pending job is {{ $value }} ms old. Workers may need scaling."
```

## Prometheus Configuration Reference

Full Prometheus configuration in the Helm chart:

| Value | Description | Default |
| - | - | - |
| `prometheus.chartManaged` | Manage Prometheus via this chart | `true` |
| `prometheus.server.retention` | Data retention period | `60d` |
| `prometheus.server.persistentVolume.size` | Storage size | `6Gi` |
| `prometheus.server.persistentVolume.storageClass` | Storage class | `""` (default) |
| `prometheus.server.resources.requests.cpu` | CPU request | `200m` |
| `prometheus.server.resources.requests.memory` | Memory request | `512Mi` |
| `prometheus.server.resources.limits.cpu` | CPU limit | `500m` |
| `prometheus.server.resources.limits.memory` | Memory limit | `2Gi` |
| `prometheus.server.global.scrape_interval` | Scrape interval | `15s` |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.