> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Security

> Security model, encryption, secrets management, and hardening for LangWatch

Self-hosted LangWatch authenticates users, encrypts stored credentials, isolates tenants at the query layer, and hardens every pod it authors by default. Three bundled components cannot meet [strict admission control](#strict-admission-control) and must be disabled or replaced there.

## Authentication & Authorization

LangWatch handles user authentication with support for:

* Email/password (default)
* SSO providers: Azure AD, Okta, Auth0, Google, GitHub, GitLab

See [SSO Configuration](/docs/self-hosting/configuration/sso) for setup guides.

**Role-Based Access Control (RBAC)** controls what users can do within a project:

* Organization-level roles (owner, admin, member)
* Project-level permissions

**SCIM provisioning** (Enterprise) enables automated user lifecycle management from your identity provider.

**API tokens** are signed with JWT (`API_TOKEN_JWT_SECRET`) for SDK authentication.

## Encryption

### At Rest

| Data Store         | Encryption Method                                            |
| ------------------ | ------------------------------------------------------------ |
| PostgreSQL         | Provider-level encryption (RDS: AES-256, Cloud SQL: AES-256) |
| ClickHouse         | Encrypted volumes (EBS encryption, PD encryption)            |
| S3                 | Server-side encryption (SSE-S3 or SSE-KMS)                   |
| Stored credentials | Application-level encryption via `CREDENTIALS_SECRET`        |

The `CREDENTIALS_SECRET` environment variable is used to encrypt API keys and credentials stored in PostgreSQL (e.g., LLM provider keys configured in the UI). This is application-level encryption on top of database-level encryption.

### In Transit

| Path                                         | Encryption                                                |
| -------------------------------------------- | --------------------------------------------------------- |
| Client to App                                | TLS at Ingress, Load Balancer                             |
| App to PostgreSQL                            | TLS (configure via connection string: `?sslmode=require`) |
| App to ClickHouse                            | HTTPS (configure ClickHouse with TLS certificates)        |
| App to Redis                                 | TLS (configure via connection string: `rediss://...`)     |
| Inter-service (App, Workers, NLP, LangEvals) | Plain HTTP within cluster (use a service mesh for mTLS)   |

<Tip>
  For inter-service encryption, deploy a service mesh like Istio or Linkerd. This adds mTLS between all pods without application changes.
</Tip>

## Secrets Management

### Development (Auto-Generated)

For development, enable `autogen.enabled: true` in the Helm chart. This generates random secrets automatically. Not suitable for production, secrets change on reinstall.

### Production (Kubernetes Secrets)

Create secrets manually and reference them in the Helm chart:

```bash theme={null}
kubectl create secret generic langwatch-secrets \
  --namespace langwatch \
  --from-literal=credentialsEncryptionKey=$(openssl rand -hex 32) \
  --from-literal=nextAuthSecret=$(openssl rand -hex 32) \
  --from-literal=cronApiKey=$(openssl rand -hex 32)
```

Reference in `values.yaml`:

```yaml theme={null}
secrets:
  existingSecret: langwatch-secrets
```

### Production (External Secret Managers)

For tighter security, use an external secrets operator to sync secrets from your cloud provider:

* **AWS Secrets Manager**: via [External Secrets Operator](https://external-secrets.io/)
* **HashiCorp Vault**: via [Vault Secrets Operator](https://developer.hashicorp.com/vault/docs/platform/k8s/vso)
* **Azure Key Vault**: via [Azure Key Vault Provider](https://azure.github.io/secrets-store-csi-driver-provider-azure/)

The Helm chart's `secretKeyRef` pattern works with any Kubernetes Secret, regardless of how it was created.

## Network Security

### Recommended Network Architecture

* **Only the LangWatch App should be exposed externally** via Ingress or Load Balancer
* All other components (Workers, NLP, LangEvals, PostgreSQL, ClickHouse, Redis) should be on internal networks only (ClusterIP services)
* Place databases in private subnets with no internet access
* Use VPC endpoints, PrivateLink for S3 access

### Private control-plane paths are blocked at the ingress

Everything under `/api/internal/*` is the app's private control plane: the Langy
agent's callbacks and the AI gateway's usage callbacks. Those routes authenticate
with a shared secret, but they are never meant to be reachable from the internet.

The chart blocks them at the ingress by default, via `ingress.blockedPaths`
(default `["/api/internal"]`). Each blocked prefix is routed to a Service with no
endpoints, so the controller has nowhere to forward the request:

```yaml theme={null}
ingress:
  enabled: true
  # Defaults to ["/api/internal"]. Add prefixes your install keeps in-cluster.
  blockedPaths:
    - /api/internal
    - /api/cron   # only if you do NOT drive cron from an external scheduler
```

Before you change it:

* **Blocked requests answer 503** on ingress-nginx. The exact code is
  controller-dependent (the Ingress API does not specify what a backend with no
  endpoints returns), and this is verified against ingress-nginx only. If you
  run a different controller, confirm it answers 5xx rather than falling through
  to another rule before relying on the block. Expect a low background rate in
  your ingress 5xx metrics from scanners: that is the block working, not the
  app faulting.
* **In-cluster callers are unaffected.** The agent, gateway and CronJobs use the
  internal Service.
* **It covers this ingress only.** If you publish the app another way (NodePort,
  LoadBalancer, your own Ingress), restrict these paths where that route ends.
* **The chart refuses configurations that would defeat the block.** Each of
  these renders a manifest that looks protected and is not, so the chart fails
  at render time rather than shipping it:
  * an `ingress.hosts` path nested under a blocked prefix (it out-matches the
    blackhole)
  * a blocked prefix with a trailing slash, a repeated slash, or surrounding
    whitespace (none of them match any real request path)
  * a regex `pathType: ImplementationSpecific` path, which ingress-nginx ranks
    *above* prefix rules
  * the `nginx.ingress.kubernetes.io/default-backend` annotation, which is
    defined as the handler for a backend with no endpoints (exactly what the
    blackhole is), so it converts the block into a proxy
  * a host whose `http.paths` list is empty, which would leave an Ingress whose
    only rules are blackholes
* **The block depends on no principal being able to create an EndpointSlice named
  for the blackhole Service.** It is selector-less, so endpoints can only arrive by
  hand. Restrict `discovery.k8s.io/endpointslices: create` in this namespace if
  that is not already the case.
* **AWS LBC (default `target-type: instance`) and GKE without NEG** reject a
  ClusterIP backend and stop reconciling the Ingress. Set
  `alb.ingress.kubernetes.io/target-type: ip` (or the NEG equivalent), or empty
  `blockedPaths` and block at the load balancer instead.
* **To disable**, set `blockedPaths: []` in a values file or use `--set-json`;
  plain `--set` assigns a string and is rejected.
* **Gateway or Langy agent running outside the cluster?** Their callbacks arrive
  over the public ingress and are blocked. Narrow the list rather than
  emptying it.

### Kubernetes Network Policies

Restrict traffic between pods:

```yaml theme={null}
# Example: only allow app and workers to reach ClickHouse
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: clickhouse-access
  namespace: langwatch
spec:
  podSelector:
    matchLabels:
      app: clickhouse
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: app
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: workers
      ports:
        - port: 8123
```

### Firewall Rules

| Source         | Destination   | Port | Protocol |
| -------------- | ------------- | ---- | -------- |
| Internet, VPN  | App (Ingress) | 443  | HTTPS    |
| App            | PostgreSQL    | 5432 | TCP      |
| App, Workers   | ClickHouse    | 8123 | HTTP     |
| App, Workers   | Redis         | 6379 | TCP      |
| Workers        | NLP           | 5561 | HTTP     |
| Workers        | LangEvals     | 5562 | HTTP     |
| NLP, LangEvals | External LLMs | 443  | HTTPS    |
| CronJobs       | App           | 5560 | HTTP     |

## Pod Security

Every LangWatch pod (app, workers, NLP, LangEvals, gateway, and the cron pods) and every bundled datastore (PostgreSQL, Redis, ClickHouse, Keeper) runs hardened. The defaults the first-party services inherit:

```yaml theme={null}
# Pod-level (global.podSecurityContext)
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
  type: RuntimeDefault

# Container-level (global.containerSecurityContext)
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
  drop: [ALL]
```

The bundled datastores carry the same posture but keep their image's own uid: PostgreSQL and Redis run 999, ClickHouse and Keeper 101, the gateway 65532, and Keeper's init container 65534. If you write a Gatekeeper `MustRunAs` constraint, allow all of those, not only 1000: a constraint that misses the init container's 65534 denies the whole Keeper pod. (The bundled Prometheus also runs 65534, though `strict-admission` removes it.)

`runAsNonRoot` is deliberately set at both pod and container level. Kubernetes inherits the pod-level value, but some Gatekeeper constraints read the container-level field directly and deny pods that only carry it on the pod.

No LangWatch pod mounts a ServiceAccount token (`global.automountServiceAccountToken: false`); none of these services talk to the Kubernetes API. The bundled Prometheus does mount one. It is an upstream subchart, and `strict-admission` removes it.

The root filesystem is read-only on every one of those pods. Anything a process writes at runtime (the app's `/tmp`, ClickHouse's server logs and `/tmp`, Postgres' socket directory) lands on an `emptyDir` mounted over that path; persistent data stays on its PVC. The image layer is never writable.

### Strict admission control

On a cluster enforcing Pod Security Admission [`restricted`](https://kubernetes.io/docs/concepts/security/pod-security-standards/) or an equivalent Gatekeeper / Kyverno bundle, apply the [`strict-admission` overlay](https://github.com/langwatch/langwatch/blob/main/charts/langwatch/examples/overlays/strict-admission.yaml):

```bash theme={null}
helm install lw . \
  -f examples/overlays/size-prod.yaml \
  -f examples/overlays/access-ingress.yaml \
  -f examples/overlays/strict-admission.yaml
```

A **default** install does not pass on its own. Everything LangWatch authors already complies (read-only root, non-root at both levels, dropped capabilities, `RuntimeDefault` seccomp, no privilege escalation, no automounted token, CPU + memory requests and limits), but three bundled components cannot, and the overlay turns all three off:

* **Prometheus** (`prometheus.chartManaged`) is an upstream subchart LangWatch doesn't control. Its pods carry no `readOnlyRootFilesystem`, no seccomp profile, and no resource limits on the config-reload sidecar. Bring your own, or disable metrics.
* **The Langy assistant** (`langyagent.chartManaged`) runs its manager as root with `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETUID` and `SETGID`. That is by design: the manager gives every assistant worker a distinct UID, and without those capabilities sibling workers share a UID and can read each other's credentials. Don't force it non-root. Deploy it on a cluster that allows it, or run without the assistant.
* **The ClickHouse preflight Job** (`clickhouse.preflight.enabled`) shells out to `kubectl` to check your Secret's keys, so it needs both an automounted token and a writable root for kubectl's discovery cache. It only renders when you supply the ClickHouse Secret yourself rather than using autogen, which is the production path, so the overlay pins it off.

One more toggle in the overlay isn't an admission failure but an unmet dependency:

* **Gateway HPA** (`gateway.autoscaling.enabled`) scales on a custom metric via `prometheus-adapter`. Set it to `false` if you don't run one; the overlay pins `gateway.replicaCount: 2` instead.

With Prometheus off, app metrics reporting (`app.telemetry.metrics.enabled`) is off by default too, so nothing exposes or scrapes a `/metrics` endpoint; the overlay pins it off to keep the two coupled. Anonymous usage analytics (`app.telemetry.usage.enabled`) is a separate setting, on by default; set it to `false` for an air-gapped or no-egress install.

In-cluster datastores satisfy read-only-root policies, but managed databases (the `postgres-external`, `redis-external`, and `clickhouse-external` overlays) are still the better production choice: you get backups, HA, and patching, and there are no datastore pods to admit.

## PII Redaction

LangWatch includes a built-in PII redaction pipeline step that automatically detects and masks personally identifiable information in traces before storage.

* **Enabled by default** in the Helm chart
* Disable with `app.features.disablePiiRedaction: true` (not recommended)
* Runs as part of the event sourcing pipeline in workers

## Multitenancy

LangWatch enforces tenant isolation at the application level:

* Every ClickHouse query includes `WHERE TenantId = ...` as the first predicate
* PostgreSQL queries include `projectId` in WHERE clauses
* API tokens are scoped to a specific project
* Cross-tenant data access is prevented at the query layer

## Supply Chain

LangWatch container images and CLI packages are published with verifiable supply-chain attestations so operators can confirm an artifact was built by LangWatch CI from a specific source commit.

### Container images (Docker Hub)

Every release of `langwatch/langwatch`, `langwatch/langwatch_nlp`, `langwatch/langevals`, and `langwatch/ai-gateway` is signed with [Sigstore](https://www.sigstore.dev/) cosign using keyless OIDC. Both the multi-arch index manifest and each per-platform manifest (linux/amd64, linux/arm64) are signed by digest. A CycloneDX SBOM is generated per platform and attached as a cosign attestation against the matching platform manifest digest, so the SBOM you verify always corresponds to the architecture you actually pulled.

Verify a signature with [cosign](https://github.com/sigstore/cosign):

```bash theme={null}
cosign verify langwatch/langwatch:<tag> \
  --certificate-identity-regexp '^https://github\.com/langwatch/langwatch/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
```

Inspect the attached SBOM for the platform you pulled (cosign resolves the right per-platform manifest digest automatically when you pass a tag):

```bash theme={null}
cosign download attestation \
  --predicate-type https://cyclonedx.org/bom \
  langwatch/langwatch:<tag> \
  | jq -r '.payload | @base64d | fromjson | .predicate' \
  > langwatch.cdx.json
```

The per-platform `*.cdx.json` files (e.g. `langwatch-linux-amd64.cdx.json`, `langwatch-linux-arm64.cdx.json`) are also attached to each `langwatch@vX.Y.Z` [GitHub release](https://github.com/langwatch/langwatch/releases).

### npm CLI

The `langwatch` npm package is published with [npm provenance attestations](https://docs.npmjs.com/generating-provenance-statements) via GitHub Actions OIDC, also backed by Sigstore. The provenance link is visible on the [package page](https://www.npmjs.com/package/langwatch) and can be verified with `npm audit signatures`.

## Production Hardening Checklist

* [ ] `autogen.enabled: false`, use manually created secrets
* [ ] All secrets stored in a secrets manager (not inline in values.yaml)
* [ ] TLS enabled on Ingress (HTTPS only)
* [ ] Database connections use TLS (`?sslmode=require`)
* [ ] PostgreSQL, ClickHouse, Redis in private subnets (no public access)
* [ ] Network policies restrict pod-to-pod traffic
* [ ] S3 buckets have public access blocked
* [ ] ClickHouse backups enabled and tested
* [ ] Monitoring and alerting configured
* [ ] Secret rotation procedure documented
* [ ] Pod security contexts verified (non-root, read-only filesystem, `RuntimeDefault` seccomp, no automounted token)
* [ ] On strict admission clusters: apply `examples/overlays/strict-admission.yaml` (`prometheus.chartManaged: false`, `langyagent.chartManaged: false`, and, without a custom metrics API, `gateway.autoscaling.enabled: false`)
* [ ] Ingress rate limiting configured
* [ ] Audit logs enabled (Enterprise)
* [ ] Image signatures verified at pull time (admission controller or `cosign verify` in CI)