Observability & Monitoring

Monitor LangWatch infrastructure with Prometheus, Grafana, and health checks

LangWatch exposes Prometheus metrics and health check endpoints for monitoring your self-hosted deployment.

Prometheus

The Helm chart includes an optional Prometheus instance that scrapes metrics from LangWatch components.

Enable Prometheus

# In your values.yaml
app:
  telemetry:
    metrics:
      enabled: true
      apiKey:
        value: "your-metrics-api-key" # Authenticates scrape requests

prometheus:
  chartManaged: true
  server:
    retention: 30d
    persistentVolume:
      size: 20Gi

What Gets Scraped

Component	Port	Endpoint	Metrics
App	5560	/metrics	Trace ingestion, ClickHouse query latency, event loop lag
Workers	2999	/metrics	Queue backlog, job processing time, job success/failure rates

Each component has its own scrape job. A job keeps only pods annotated prometheus.io/scrape: "true" whose app.kubernetes.io/name label is that component’s (<release>-app or <release>-workers). The chart adds the annotation to the app and workers pods when app.telemetry.metrics.enabled is true. Neither component serves /metrics until app.telemetry.metrics.enabled is on with an API key, and then both require that key as a bearer token.

The chart-managed Prometheus sends the key only when it is set through app.telemetry.metrics.apiKey.value. With apiKey.secretKeyRef, the pods get the key but the scrape config does not, and both jobs are answered with 401.

Access Prometheus

Port-forward to the Prometheus UI:

kubectl -n langwatch port-forward svc/langwatch-prometheus-server 9090:9090
# Open http://localhost:9090

External Prometheus

To use an existing Prometheus instance instead of the chart-managed one:

prometheus:
  chartManaged: false
  external:
    existingSecret: prometheus-credentials
    secretKeys:
      host: "host"
      port: "port"
      username: "username"
      password: "password"

You’ll need to configure your external Prometheus to scrape the LangWatch pods, sending the metrics API key as a bearer token. With app.telemetry.metrics.enabled: true, the app pods are annotated with:

prometheus.io/scrape: "true"
prometheus.io/port: "5560"

The workers pods are annotated with:

prometheus.io/scrape: "true"
prometheus.io/port: "2999"

Both serve the metrics on the path /metrics.

Grafana

Connect Grafana to your Prometheus instance to visualize LangWatch metrics.

Key Dashboards

Set up dashboards for:

• Span throughput: spans ingested per minute
• Worker queue backlog: groups waiting to be processed and the age of the oldest pending job (indicates processing bottleneck)
• ClickHouse query latency: p50/p95/p99 query times
• Resource utilization: CPU and memory per component

Example Queries

# Span ingestion rate (per minute), by outcome
sum by (outcome) (rate(trace_ingestion_spans_total[5m])) * 60

# Worker queue backlog
sum by (queue_name) (gq_pending_groups)

# Age of the oldest due job, in seconds, including groups held in retry backoff
max by (queue_name) (gq_oldest_backlog_age_milliseconds) / 1000

# ClickHouse query p95 latency
histogram_quantile(0.95, sum by (le) (rate(clickhouse_query_duration_seconds_bucket[5m])))

Health & Readiness Checks

Endpoints

Component	Endpoint	Method	Healthy Response
App	/api/health	GET	200 OK
Workers	/healthz	GET	200 OK

Kubernetes Probes

The Helm chart configures probes automatically. The app probes /api/health on port 5560:

# Startup probe (allows up to 40 minutes: on a first install the app runs the upgrade before it serves)
startupProbe:
  httpGet:
    path: /api/health
    port: http
  periodSeconds: 15
  failureThreshold: 160

# Readiness probe (removes from service if unhealthy)
readinessProbe:
  httpGet:
    path: /api/health
    port: http
  periodSeconds: 10
  timeoutSeconds: 5

# Liveness probe (restarts unhealthy pods)
livenessProbe:
  httpGet:
    path: /api/health
    port: http
  periodSeconds: 30
  timeoutSeconds: 5

The workers use the same startup budget and a liveness probe every 30 seconds, both on /healthz at port 2999. They have no readiness probe.

Manual Health Check

# Check app health
kubectl -n langwatch exec deploy/langwatch-app -- \
  curl -s http://localhost:5560/api/health

# Check worker health
kubectl -n langwatch exec deploy/langwatch-workers -- \
  curl -s http://localhost:2999/healthz

Alerting Recommendations

Set up alerts for these critical conditions:

Alert	Condition	Severity
Worker queue backlog	Oldest pending job older than 5 min	Warning
Worker queue backlog (critical)	Oldest pending job older than 30 min	Critical
ClickHouse memory	Memory usage > 80% of limit	Warning
ClickHouse disk	Hot storage > 85% full	Critical
PostgreSQL connections	Active connections > 80% of max	Warning
App error rate	HTTP 5xx rate > 5% for 5 min	Critical
Pod restarts	Pod restart count > 3 in 15 min	Warning

Example Alertmanager Rule

groups:
  - name: langwatch
    rules:
      - alert: WorkerQueueBacklog
        expr: max by (queue_name) (gq_oldest_backlog_age_milliseconds) > 300000
        labels:
          severity: warning
        annotations:
          summary: "Worker queue backlog is high"
          description: "Oldest pending job is {{ $value }} ms old. Workers may need scaling."

Prometheus Configuration Reference

Full Prometheus configuration in the Helm chart:

Value	Description	Default
prometheus.chartManaged	Manage Prometheus via this chart	true
prometheus.server.retention	Data retention period	60d
prometheus.server.persistentVolume.size	Storage size	6Gi
prometheus.server.persistentVolume.storageClass	Storage class	"" (default)
prometheus.server.resources.requests.cpu	CPU request	200m
prometheus.server.resources.requests.memory	Memory request	512Mi
prometheus.server.resources.limits.cpu	CPU limit	500m
prometheus.server.resources.limits.memory	Memory limit	2Gi
prometheus.server.global.scrape_interval	Scrape interval	15s