Architecture & Infrastructure

How LangWatch components fit together, what you’re deploying and how data flows through the system

A self-hosted LangWatch deployment runs the app, the workers, the AI Gateway, the Langy agent, PostgreSQL, Redis, ClickHouse with S3 storage, and the NLP and evaluation services, connected by an event-sourcing pipeline. The langwatch Helm chart deploys all of them; the AI Gateway and the Langy agent come from its gateway and langyagent subcharts.

System Overview

Every component runs in your environment. The dashed line is the only connection to LangWatch, and each part of it has a switch.

API Layer

LangWatch App (:5560)

Web UI, REST API, OTel ingestion, authentication, SSE real-time updates. The main externally exposed component; the AI Gateway can have its own ingress.

Processing

LangWatch Workers

Event sourcing pipeline via BullMQ, span ingestion, trace summarisation, cost enrichment, evaluations, PII redaction, and more.

Services

LangWatch NLP (:5561) and LangEvals (:5562)

Workflow execution and built-in evaluators, guardrails. Call external LLMs for model-based operations.

AI Gateway

AI Gateway (:5563)

OpenAI-compatible proxy for virtual keys: routing, fallbacks, budgets, guardrails. Calls the model providers you configure.

Assistant

Langy Agent (:8080)

Runs the in-product assistant’s workers. Internal only, reached by the App and Workers.

Control Plane

PostgreSQL

Users, teams, projects, configurations, prompt versions. Managed via Prisma; migrations are applied by the upgrade before the app starts.

Data Plane

ClickHouse

All traces, spans, evaluations, experiments, analytics. Hot storage on SSD, cold storage on S3. Auto-tuned via the clickhouse-serverless subchart.

Queue & Storage

Redis: BullMQ job queue, caching, sessions.

S3: ClickHouse cold storage, backups, datasets.

Components

LangWatch App (port 5560)

The API process (apps/api, built on Hono), the external entry point for everything except AI Gateway traffic. It serves REST, tRPC and SSE, and also serves the prebuilt browser bundle (a Vite/React single-page app, apps/ui) as static assets. One container is the whole interactive tier:

• Web UI: dashboards, trace explorer, prompt management, experiment views
• REST API + OTel ingestion: receives spans from LangWatch SDKs
• Authentication: email, Google, GitHub, GitLab, Azure AD, Auth0, Okta
• SSE: pushes real-time updates to connected browser clients
• Analytics queries: reads from ClickHouse
• Control plane: manages users, teams, projects via PostgreSQL

LangWatch Workers

Same langwatch/langwatch image, started from apps/worker instead of apps/api (there is no in-process worker mode: the app and the workers are always separate processes). Consumes jobs from a BullMQ queue in Redis and runs the event sourcing pipeline (see below).

• Deployed as a separate Kubernetes Deployment
• Stateless, scale by adding replicas

LangWatch NLP (port 5561)

Go service (services/nlpgo, shipped as the langwatch_nlp image) for:

• Optimization Studio workflow execution, synchronous and streamed
• The prompt playground’s model calls, proxied to the configured provider

Topic clustering is not in this service: the Workers schedule it and send the clustering batches to LangEvals.

LangEvals (port 5562)

Built-in evaluator library (Python):

• LLM-as-a-Judge (boolean, categorical, scored)
• Safety (content safety, jailbreak detection, prompt shield)
• Quality (faithfulness, relevancy, correctness, summarization)
• RAG (context precision, context recall, context relevancy)
• Format (exact match, BLEU, ROUGE, semantic similarity)

Both NLP and LangEvals make outbound calls to external LLM providers for model-based operations.

AI Gateway (port 5563)

A Go service (image langwatch/ai-gateway, subchart gateway) that terminates virtual-key traffic and forwards it to the model providers configured in LangWatch. It reads each key’s configuration (providers, budgets, guardrails, routing) from the App over the cluster network and reports usage back to it.

• Your LLM clients, coding agents and the Langy agent call it with a virtual key
• It needs its own externally reachable hostname (ingress.gateway.host, or gateway.publicUrl behind your own route) when clients outside the cluster use it
• gateway.chartManaged: false removes it

Langy Agent (port 8080, Service port 80)

The worker manager behind the in-product assistant (image langwatch/langyagent, subchart langyagent). The App and Workers reach it over cluster DNS; it has no ingress. Each conversation runs in a worker with its own UID and credentials, and its model calls go through the AI Gateway.

• On by default (langyagent.chartManaged: true); false removes the pod and every reference to it
• Runs on the node’s normal container runtime by default; a sandboxed RuntimeClass (gVisor, Kata) is opt-in, see Langy setup
• Its NetworkPolicy denies internet egress unless you turn it on, see Langy networking and egress

PostgreSQL: Control Plane

Stores users, teams, projects, configurations, prompt versions, evaluator definitions. Managed via Prisma ORM; pnpm task upgrade applies its migrations before the app and workers start, and they refuse to start until it has.

ClickHouse: Data Plane

Stores all high-volume data: traces, spans, evaluations, experiments, analytics, and event sourcing events/projections.

Mode	Replicas	Engine	Use Case
Standalone	1	MergeTree	Dev, small production
Replicated	3+ (odd)	ReplicatedMergeTree + Keeper	HA production

The clickhouse-serverless subchart auto-tunes internal parameters from two inputs: cpu and memory.

The langwatch/clickhouse-serverless Docker image is a performance-tweaked ClickHouse build optimized for LangWatch’s traffic patterns, high-throughput event ingestion with concurrent analytical queries.

Tiered storage: hot data on local SSD, cold data on S3 after a configurable TTL (default 49 days). Native BACKUP/RESTORE to S3.

Redis

• BullMQ job queue: connects the App to Workers with guaranteed delivery, retry, and backpressure
• Caching: frequently accessed config and lookup data
• Sessions: user session storage

S3, Object Storage

• ClickHouse cold storage (tiered after TTL)
• ClickHouse backups (full + incremental)
• Dataset storage (optional)
• Stored objects: externalized byte content (audio/image/video/document) content-addressed under {projectId}/{sha256} and served back via GET /api/v1/files/:id.

Event Sourcing Pipeline

LangWatch v3 uses an event-sourcing model for data processing. Understanding this helps with debugging and capacity planning.

When a span arrives from your SDK, it enters a pipeline of independent steps running on the Workers:

Each step reads from the queue, does its work, and writes results to ClickHouse. Steps are independent, if one is slow (e.g., evaluation waiting on an LLM call), others continue processing.

How Data is Organized

The pipeline produces three types of output:

Events: immutable records of what happened. Stored in ClickHouse and never modified.

Event	Produced By	What It Contains
SpanIngested	Span Ingestion	Raw span data from SDK
TraceSummarised	Trace Summarisation	Aggregated trace with input/output
CostEnriched	LLM Cost Enrichment	Token costs per model
MetricsExtracted	LLM Metric Processing	Latency, token counts, model info
EmbeddingsGenerated	Embedding Extraction	Vector embeddings for similarity
PIIRedacted	PII Redaction	Redacted fields and detection metadata
EvaluationCompleted	Evaluation Execution	Evaluator scores and results
ExperimentResultRecorded	Experiment Processing	Run results for A/B tests

Projections: derived tables that dashboards and APIs read from. Built from events.

Projection	Built From	Used By
Traces	SpanIngested + TraceSummarised	Trace explorer, search
Spans	SpanIngested	Span detail views
Evaluations	EvaluationCompleted	Quality scores, monitors
ExperimentRuns	ExperimentResultRecorded	Experiment result tables
Analytics	All events	Dashboard aggregations
Topics	TraceSummarised + clustering	Conversation topic groups

Reactions: side effects triggered during processing.

Reaction	Triggered By	Effect
SSE Update	Any event	Real-time UI refresh in browser
Alert, Trigger	EvaluationCompleted	Slack, email, webhook notification
Dataset Append	Automation rules	Auto-add traces to datasets

If Worker queue depth grows in Redis, it means processing is falling behind ingestion. The fix is to add more Worker replicas, each one is stateless and consumes jobs independently.

Data Flow

The path of one trace. PII redaction runs on the Workers before the trace is written to ClickHouse.

1. Your application sends spans to the App over OTLP or REST.
2. The App enqueues a span ingestion job in Redis (BullMQ).
3. A Worker picks up the job.
4. The Worker runs the pipeline steps (enrichment, PII redaction) and writes events and projections to ClickHouse.
5. When evaluators are configured, the Worker requests the evaluation from LangEvals and writes the scores to ClickHouse.
6. The Worker broadcasts the change. The App reads ClickHouse and pushes the update to connected browsers over SSE.
7. After the hot-storage TTL, ClickHouse moves data to object storage.

Periodic work (topic clustering, alert triggers, webhook delivery log pruning) is scheduled on the Workers. The chart ships no Kubernetes CronJobs; cronjobs.jobs stays empty unless you add your own.

Network Topology

The App is exposed externally. The AI Gateway is exposed only when you set ingress.gateway.host, which renders a second Ingress for /v1 and /health with the app Ingress class. See Routing. Everything else is cluster-internal:

Component	Service Type	External
App	ClusterIP (app.service.type), plus an Ingress when ingress.enabled is set	Yes
Workers	None (no Service needed)	No
NLP	ClusterIP	No
LangEvals	ClusterIP	No
AI Gateway	ClusterIP, plus an Ingress when ingress.gateway.host is set	Optional, for clients outside the cluster
Langy Agent	ClusterIP	No
PostgreSQL	ClusterIP	No
ClickHouse	ClusterIP	No
Redis	ClusterIP	No

LangEvals, NLP and the AI Gateway make outbound calls to external LLM providers (OpenAI, Azure, etc.). Ensure these pods have network egress to the relevant endpoints. For the full list of outbound hosts, see Data and telemetry.

Connected agents

A connected agent is the customer’s own process holding an outbound connection to the App, a WebSocket by default with HTTP long polling as the fallback. The first three requirements apply to the WebSocket transport, the last one to long polling:

• WebSocket upgrade on /api/v1/agents/connect. The App serves it on its normal HTTP listener, next to /api/trpc-ws. An ingress that already passes /api/trpc-ws needs no change; one that terminates or rewrites upgrades must pass this path through.
• A read timeout above 15 seconds. The App pings every 15 seconds to keep the connection alive. A shorter idle timeout closes the connection between pings, and the agent flaps between Online and Offline. The access-ingress.yaml overlay sets proxy-read-timeout: 120.
• Outbound WSS from the customer’s side. The process that runs the agent opens wss:// to your App’s public hostname. The customer’s side opens no listening port and needs no inbound rule.
• HTTP long polling for a network that blocks WebSockets. The same frames also travel over POST /api/v1/agents/connect/register, GET /api/v1/agents/connect/poll and POST /api/v1/agents/connect/frames. A poll is held open for up to 25 seconds, so the ingress read timeout must be above that; the proxy-read-timeout: 120 above covers it. The SDK switches to this path on its own when a proxy answers the upgrade with an HTTP status, or from the start with LANGWATCH_AGENT_TRANSPORT=http.

Redis is required for more than one App replica. Instance presence and call dispatch live in Redis; without it the registry is in memory, correct for one replica only, and the App refuses a connection when LANGWATCH_APP_REPLICAS is above 1.

That variable defaults to 1, and Helm scales the App with app.replicaCount on its own, so a Helm deployment must set LANGWATCH_APP_REPLICAS to the same value as app.replicaCount.

Without Redis, a mismatch fails in one of two ways. If app.replicaCount is above LANGWATCH_APP_REPLICAS, each pod runs its own in-memory registry, and an agent connected to one pod reads as Offline on the others. If LANGWATCH_APP_REPLICAS is above app.replicaCount, the App refuses every agent connection, also when only one pod runs.

Docker Images

Image	Port	Purpose
langwatch/langwatch	5560	App + Workers (same image, different entrypoint)
langwatch/langwatch_nlp	5561	NLP engine (Go): workflows, playground proxy
langwatch/langevals	5562	Evaluators, guardrails
langwatch/ai-gateway	5563	AI Gateway
langwatch/langyagent	8080	Langy assistant agent

OpenTelemetry Integration

LangWatch is deeply integrated with OpenTelemetry. The platform both consumes and exports telemetry data:

Ingestion: The LangWatch App accepts spans via the OpenTelemetry protocol (OTLP over HTTP). Any OTel-instrumented application can send traces to LangWatch without a vendor-specific SDK.

Export: LangWatch exports its own operational metrics, logs, and traces via OpenTelemetry for infrastructure debugging:

• Metrics: Prometheus-compatible metrics from the App and Workers (request latency, queue depth, error rates)
• Logs: Structured application logs from all components
• Traces: Distributed traces of internal request processing

This means you can monitor LangWatch itself using the same observability stack you use for the rest of your infrastructure, Grafana, Datadog, New Relic, or any OTel-compatible backend.

LangWatch ships with off-the-shelf Grafana dashboards for monitoring the platform. See Observability & Monitoring for setup details.

Deployment Models

Self-Managed

Everything on your infrastructure. You deploy the Helm chart and manage all components.

Cloud Enterprise

LangWatch manages the control plane in a dedicated, single-tenant environment. Exclusive data instances in your preferred region.

Hybrid (Bring Your Own Storage)

LangWatch manages compute (App, Workers, NLP, LangEvals, AI Gateway). You bring your own ClickHouse + S3 in your VPC.

For Cloud Enterprise or Hybrid, contact the LangWatch team.