> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Architecture & Infrastructure

> How LangWatch components fit together, what you're deploying and how data flows through the system

A self-hosted LangWatch deployment runs the app, the workers, the AI Gateway, the Langy agent, PostgreSQL, Redis, ClickHouse with S3 storage, and the NLP and evaluation services, connected by an event-sourcing pipeline. The `langwatch` Helm chart deploys all of them; the AI Gateway and the Langy agent come from its `gateway` and `langyagent` subcharts.

## System Overview

![Architecture of a self-hosted LangWatch install: agents, coding tools and the team on the left, the Kubernetes cluster with the LangWatch services and data stores in the middle, model providers on the right, and one optional dashed connection to LangWatch cloud](https://mintcdn.com/langwatch/ZJQMrSTiD9Am6jr0/images/self-hosting/self-hosting-overview-light.svg?fit=max\&auto=format\&n=ZJQMrSTiD9Am6jr0\&q=85\&s=4c2c8f4e27b33aeb58dece565254fa56)

Every component runs in your environment. The dashed line is the only connection to LangWatch, and each part of it has a switch.

> [card] API Layer
>
> **LangWatch App** (:5560)
>
> Web UI, REST API, OTel ingestion, authentication, SSE real-time updates. The main externally exposed component; the AI Gateway can have its own ingress.

> [card] Processing
>
> **LangWatch Workers**
>
> Event sourcing pipeline via BullMQ, span ingestion, trace summarisation, cost enrichment, evaluations, PII redaction, and more.

> [card] Services
>
> **LangWatch NLP** (:5561) and **LangEvals** (:5562)
>
> Workflow execution and built-in evaluators, guardrails. Call external LLMs for model-based operations.

> [card] AI Gateway
>
> **AI Gateway** (:5563)
>
> OpenAI-compatible proxy for virtual keys: routing, fallbacks, budgets, guardrails. Calls the model providers you configure.

> [card] Assistant
>
> **Langy Agent** (:8080)
>
> Runs the in-product assistant's workers. Internal only, reached by the App and Workers.

> [card] Control Plane
>
> **PostgreSQL**
>
> Users, teams, projects, configurations, prompt versions. Managed via Prisma; migrations are applied by the upgrade before the app starts.

> [card] Data Plane
>
> **ClickHouse**
>
> All traces, spans, evaluations, experiments, analytics. Hot storage on SSD, cold storage on S3. Auto-tuned via the clickhouse-serverless subchart.

> [card] Queue & Storage
>
> **Redis**: BullMQ job queue, caching, sessions.
>
> **S3**: ClickHouse cold storage, backups, datasets.

## Components

### LangWatch App (port 5560)

The API process (`apps/api`, built on Hono), the external entry point
for everything except AI Gateway traffic. It serves REST, tRPC and SSE, and also serves the prebuilt
browser bundle (a Vite/React single-page app, `apps/ui`) as static assets.
One container is the whole interactive tier:

* **Web UI**: dashboards, trace explorer, prompt management, experiment views
* **REST API + OTel ingestion**: receives spans from LangWatch SDKs
* **Authentication**: email, Google, GitHub, GitLab, Azure AD, Auth0, Okta
* **SSE**: pushes real-time updates to connected browser clients
* **Analytics queries**: reads from ClickHouse
* **Control plane**: manages users, teams, projects via PostgreSQL

### LangWatch Workers

Same `langwatch/langwatch` image, started from `apps/worker` instead of
`apps/api` (there is no in-process worker mode: the app and the workers are
always separate processes). Consumes jobs from a BullMQ queue in Redis and
runs the event sourcing pipeline (see below).

* Deployed as a separate Kubernetes Deployment
* Stateless, scale by adding replicas

### LangWatch NLP (port 5561)

Go service (`services/nlpgo`, shipped as the `langwatch_nlp` image) for:

* Optimization Studio workflow execution, synchronous and streamed
* The prompt playground's model calls, proxied to the configured provider

Topic clustering is not in this service: the Workers schedule it and send the clustering batches to LangEvals.

### LangEvals (port 5562)

Built-in evaluator library (Python):

* LLM-as-a-Judge (boolean, categorical, scored)
* Safety (content safety, jailbreak detection, prompt shield)
* Quality (faithfulness, relevancy, correctness, summarization)
* RAG (context precision, context recall, context relevancy)
* Format (exact match, BLEU, ROUGE, semantic similarity)

Both NLP and LangEvals make outbound calls to external LLM providers for model-based operations.

### AI Gateway (port 5563)

A Go service (image `langwatch/ai-gateway`, subchart `gateway`) that terminates virtual-key traffic and forwards it to the model providers configured in LangWatch. It reads each key's configuration (providers, budgets, guardrails, routing) from the App over the cluster network and reports usage back to it.

* Your LLM clients, coding agents and the Langy agent call it with a virtual key
* It needs its own externally reachable hostname (`ingress.gateway.host`, or `gateway.publicUrl` behind your own route) when clients outside the cluster use it
* `gateway.chartManaged: false` removes it

### Langy Agent (port 8080, Service port 80)

The worker manager behind the in-product assistant (image `langwatch/langyagent`, subchart `langyagent`). The App and Workers reach it over cluster DNS; it has no ingress. Each conversation runs in a worker with its own UID and credentials, and its model calls go through the AI Gateway.

* On by default (`langyagent.chartManaged: true`); `false` removes the pod and every reference to it
* Runs on the node's normal container runtime by default; a sandboxed RuntimeClass (gVisor, Kata) is opt-in, see [Langy setup](https://langwatch.ai/docs/self-hosting/langy/setup.md)
* Its NetworkPolicy denies internet egress unless you turn it on, see [Langy networking and egress](https://langwatch.ai/docs/self-hosting/langy/networking-and-egress.md)

### PostgreSQL: Control Plane

Stores users, teams, projects, configurations, prompt versions, evaluator definitions. Managed via Prisma ORM; `pnpm task upgrade` applies its migrations before the app and workers start, and they refuse to start until it has.

### ClickHouse: Data Plane

Stores all high-volume data: traces, spans, evaluations, experiments, analytics, and event sourcing events/projections.

| Mode | Replicas | Engine | Use Case |
| - | - | - | - |
| Standalone | 1 | MergeTree | Dev, small production |
| Replicated | 3+ (odd) | ReplicatedMergeTree + Keeper | HA production |

The `clickhouse-serverless` subchart auto-tunes internal parameters from two inputs: `cpu` and `memory`.

The `langwatch/clickhouse-serverless` Docker image is a performance-tweaked ClickHouse build optimized for LangWatch's traffic patterns, high-throughput event ingestion with concurrent analytical queries.

**Tiered storage**: hot data on local SSD, cold data on S3 after a configurable TTL (default 49 days). Native `BACKUP`/`RESTORE` to S3.

### Redis

* **BullMQ job queue**: connects the App to Workers with guaranteed delivery, retry, and backpressure
* **Caching**: frequently accessed config and lookup data
* **Sessions**: user session storage

### S3, Object Storage

* ClickHouse cold storage (tiered after TTL)
* ClickHouse backups (full + incremental)
* Dataset storage (optional)
* **Stored objects**: externalized byte content (audio/image/video/document) content-addressed under `{projectId}/{sha256}` and served back via `GET /api/v1/files/:id`.

## Event Sourcing Pipeline

LangWatch v3 uses an event-sourcing model for data processing. Understanding this helps with debugging and capacity planning.

When a span arrives from your SDK, it enters a pipeline of independent steps running on the Workers:

```mermaid theme={null}
graph LR
    subgraph Ingestion
        A[Span Ingestion] --> B[Trace Summarisation]
    end

    subgraph Enrichment
        B --> C[LLM Cost Enrichment]
        B --> D[LLM Metric Processing]
        B --> E[Embedding Extraction]
        B --> F[PII Redaction]
    end

    subgraph Evaluation
        C --> G[Evaluation Execution]
        D --> G
        G --> H[Annotation Processing]
        G --> I[Experiment Processing]
    end

    subgraph Reactions
        I --> J[Generate Topics]
        I --> K[Automation / Triggers]
        I --> L[UI Update Broadcasting]
    end

    L --> M[(ClickHouse)]
    K --> M
    J --> M
```

Each step reads from the queue, does its work, and writes results to ClickHouse. Steps are independent, if one is slow (e.g., evaluation waiting on an LLM call), others continue processing.

### How Data is Organized

The pipeline produces three types of output:

**Events**: immutable records of what happened. Stored in ClickHouse and never modified.

| Event | Produced By | What It Contains |
| - | - | - |
| SpanIngested | Span Ingestion | Raw span data from SDK |
| TraceSummarised | Trace Summarisation | Aggregated trace with input/output |
| CostEnriched | LLM Cost Enrichment | Token costs per model |
| MetricsExtracted | LLM Metric Processing | Latency, token counts, model info |
| EmbeddingsGenerated | Embedding Extraction | Vector embeddings for similarity |
| PIIRedacted | PII Redaction | Redacted fields and detection metadata |
| EvaluationCompleted | Evaluation Execution | Evaluator scores and results |
| ExperimentResultRecorded | Experiment Processing | Run results for A/B tests |

**Projections**: derived tables that dashboards and APIs read from. Built from events.

| Projection | Built From | Used By |
| - | - | - |
| Traces | SpanIngested + TraceSummarised | Trace explorer, search |
| Spans | SpanIngested | Span detail views |
| Evaluations | EvaluationCompleted | Quality scores, monitors |
| ExperimentRuns | ExperimentResultRecorded | Experiment result tables |
| Analytics | All events | Dashboard aggregations |
| Topics | TraceSummarised + clustering | Conversation topic groups |

**Reactions**: side effects triggered during processing.

| Reaction | Triggered By | Effect |
| - | - | - |
| SSE Update | Any event | Real-time UI refresh in browser |
| Alert, Trigger | EvaluationCompleted | Slack, email, webhook notification |
| Dataset Append | Automation rules | Auto-add traces to datasets |

> [!NOTE]
>
> If Worker queue depth grows in Redis, it means processing is falling behind ingestion. The fix is
> to add more Worker replicas, each one is stateless and consumes jobs independently.

## Data Flow

![The path of one trace: agent, App, Redis queue, Workers, ClickHouse and object storage, with LangEvals and the model provider below](https://mintcdn.com/langwatch/ZJQMrSTiD9Am6jr0/images/self-hosting/self-hosting-trace-flow-light.svg?fit=max\&auto=format\&n=ZJQMrSTiD9Am6jr0\&q=85\&s=c8b162d1cd19652876af0f7949a8d31b)

The path of one trace. PII redaction runs on the Workers before the trace is written to ClickHouse.

1. Your application sends spans to the App over OTLP or REST.
2. The App enqueues a span ingestion job in Redis (BullMQ).
3. A Worker picks up the job.
4. The Worker runs the pipeline steps (enrichment, PII redaction) and writes events and projections to ClickHouse.
5. When evaluators are configured, the Worker requests the evaluation from LangEvals and writes the scores to ClickHouse.
6. The Worker broadcasts the change. The App reads ClickHouse and pushes the update to connected browsers over SSE.
7. After the hot-storage TTL, ClickHouse moves data to object storage.

Periodic work (topic clustering, alert triggers, webhook delivery log pruning) is scheduled on the Workers. The chart ships no Kubernetes CronJobs; `cronjobs.jobs` stays empty unless you add your own.

## Network Topology

The App is exposed externally. The AI Gateway is exposed only when you set `ingress.gateway.host`, which renders a second Ingress for `/v1` and `/health` with the app Ingress class. See [Routing](https://langwatch.ai/docs/self-hosting/deployment/routing.md). Everything else is cluster-internal:

| Component | Service Type | External |
| - | - | - |
| App | ClusterIP (`app.service.type`), plus an Ingress when `ingress.enabled` is set | Yes |
| Workers | None (no Service needed) | No |
| NLP | ClusterIP | No |
| LangEvals | ClusterIP | No |
| AI Gateway | ClusterIP, plus an Ingress when `ingress.gateway.host` is set | Optional, for clients outside the cluster |
| Langy Agent | ClusterIP | No |
| PostgreSQL | ClusterIP | No |
| ClickHouse | ClusterIP | No |
| Redis | ClusterIP | No |

> [!NOTE]
>
> LangEvals, NLP and the AI Gateway make outbound calls to external LLM providers (OpenAI, Azure, etc.). Ensure these pods have network egress to the relevant endpoints. For the full list of outbound hosts, see [Data and telemetry](https://langwatch.ai/docs/self-hosting/data-and-telemetry.md#every-outbound-connection).

### Connected agents

A [connected agent](https://langwatch.ai/docs/agent-testing/connect-your-agent.md) is the customer's own process holding an outbound connection to the App, a WebSocket by default with HTTP long polling as the fallback. The first three requirements apply to the WebSocket transport, the last one to long polling:

* **WebSocket upgrade on `/api/v1/agents/connect`.** The App serves it on its normal HTTP listener, next to `/api/trpc-ws`. An ingress that already passes `/api/trpc-ws` needs no change; one that terminates or rewrites upgrades must pass this path through.
* **A read timeout above 15 seconds.** The App pings every 15 seconds to keep the connection alive. A shorter idle timeout closes the connection between pings, and the agent flaps between Online and Offline. The `access-ingress.yaml` overlay sets `proxy-read-timeout: 120`.
* **Outbound WSS from the customer's side.** The process that runs the agent opens `wss://` to your App's public hostname. The customer's side opens no listening port and needs no inbound rule.
* **HTTP long polling for a network that blocks WebSockets.** The same frames also travel over `POST /api/v1/agents/connect/register`, `GET /api/v1/agents/connect/poll` and `POST /api/v1/agents/connect/frames`. A poll is held open for up to 25 seconds, so the ingress read timeout must be above that; the `proxy-read-timeout: 120` above covers it. The SDK switches to this path on its own when a proxy answers the upgrade with an HTTP status, or from the start with `LANGWATCH_AGENT_TRANSPORT=http`.

Redis is required for more than one App replica. Instance presence and call dispatch live in Redis; without it the registry is in memory, correct for one replica only, and the App refuses a connection when `LANGWATCH_APP_REPLICAS` is above 1.

That variable defaults to `1`, and Helm scales the App with `app.replicaCount` on its own, so a Helm deployment must set `LANGWATCH_APP_REPLICAS` to the same value as `app.replicaCount`.

Without Redis, a mismatch fails in one of two ways. If `app.replicaCount` is above `LANGWATCH_APP_REPLICAS`, each pod runs its own in-memory registry, and an agent connected to one pod reads as Offline on the others. If `LANGWATCH_APP_REPLICAS` is above `app.replicaCount`, the App refuses every agent connection, also when only one pod runs.

## Docker Images

| Image | Port | Purpose |
| - | - | - |
| `langwatch/langwatch` | 5560 | App + Workers (same image, different entrypoint) |
| `langwatch/langwatch_nlp` | 5561 | NLP engine (Go): workflows, playground proxy |
| `langwatch/langevals` | 5562 | Evaluators, guardrails |
| `langwatch/ai-gateway` | 5563 | AI Gateway |
| `langwatch/langyagent` | 8080 | Langy assistant agent |

## OpenTelemetry Integration

LangWatch is deeply integrated with OpenTelemetry. The platform both **consumes** and **exports** telemetry data:

**Ingestion**: The LangWatch App accepts spans via the OpenTelemetry protocol (OTLP over HTTP). Any OTel-instrumented application can send traces to LangWatch without a vendor-specific SDK.

**Export**: LangWatch exports its own operational metrics, logs, and traces via OpenTelemetry for infrastructure debugging:

* **Metrics**: Prometheus-compatible metrics from the App and Workers (request latency, queue depth, error rates)
* **Logs**: Structured application logs from all components
* **Traces**: Distributed traces of internal request processing

This means you can monitor LangWatch itself using the same observability stack you use for the rest of your infrastructure, Grafana, Datadog, New Relic, or any OTel-compatible backend.

LangWatch ships with off-the-shelf Grafana dashboards for monitoring the platform. See [Observability & Monitoring](https://langwatch.ai/docs/self-hosting/configuration/observability.md) for setup details.

## Deployment Models

### Self-Managed

Everything on your infrastructure. You deploy the Helm chart and manage all components.

### Cloud Enterprise

LangWatch manages the control plane in a dedicated, single-tenant environment. Exclusive data instances in your preferred region.

### Hybrid (Bring Your Own Storage)

LangWatch manages compute (App, Workers, NLP, LangEvals, AI Gateway). You bring your own ClickHouse + S3 in your VPC.

For Cloud Enterprise or Hybrid, [contact the LangWatch team](https://langwatch.ai/get-a-demo).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.