Upgrade Guide

How to upgrade LangWatch to the latest version

Before You Upgrade

1. Check release notes for breaking changes at github.com/langwatch/langwatch/releases
2. Back up your databases: see Backups
3. Test in a staging environment before upgrading production
4. Keep CREDENTIALS_SECRET unchanged. A new value makes stored provider keys and integration secrets unreadable and invalidates API keys. To replace it, follow Rotating the credentials secret as a separate change, not during an upgrade
5. Write the version 2 queue format first. This release reads only version 2 GroupQueue jobs and drops a job an older release wrote in the earlier format. Before upgrading, set GROUP_QUEUE_ENVELOPE_WRITES_ENABLED=true on the running release and roll it out, or pause traffic and let the queue drain. After the upgrade, remove that variable and ES_FOLD_TRUST_ABSENT_MISS: this release reads neither

Database privileges

The upgrade keeps its ledger in its own Postgres schema, <schema>_upgrade_ledger, beside the installation’s. The DATABASE_URL role needs CREATE on the database to create it, or you can create that schema beforehand, owned by the role. Without either, the api and the worker refuse to start and name the missing privilege.

Automatic identity and access migration

Identity and access migrations run automatically after each upgrade, organization by organization: the preparation step runs them right after pnpm task upgrade, and the workers run them again every hour. A tenant whose authorization, identifier, and existing SSO data migrate cleanly is switched over. No preparatory release or manual migration pass is required.

A tenant whose historical data disagrees with the migrated result is held on its existing legacy sign-in path: the migration leaves that tenant’s data untouched, and the app still starts and serves it as before. A tenant whose migration throws is parked the same way, also without blocking startup.

Both are retried automatically. The next upgrade and the workers’ hourly pass re-drive them, and an operator can also trigger a pass, or target one organization, without waiting. See Check the migration results below.

The preparation step still fails in three cases: the migration runner itself fails outright, the queue behind it fails to drain, or the fleet makes no progress for 25 passes (about two minutes: for example because Redis is unreachable, or another process holds every claim).

In any of them it exits non-zero, which fails the upgrade (the Helm pre-upgrade Job where one renders, otherwise the workers), and the new api is not ready until the installation is current, so the previous version keeps serving; the next run retries from scratch.

Keep the current identity-provider configuration, callback URLs, and CREDENTIALS_SECRET during the upgrade. The migration preserves the existing SSO route; switching providers remains a separate operation.

Take a fresh database backup and deploy normally. The diagnostics below help resolve held or parked tenants; they are not a routine preflight.

Check the migration results

Sign in with an operator account from ADMIN_EMAILS and open Ops > Upgrades > Tenant migrations (/ops/upgrades?tab=tenants). It lists every registered tenant migration with its Finalized, Held and Parked counts, and lets you search for an organization to inspect its status, target a migration run for it, or start a fresh pass immediately rather than waiting for the next restart.

The live connection indexes deliberately refuse duplicate slots rather than silently choosing a connection. REJECTED, DRAFT, and CLAIMED still occupy slots; only DISCARDED and TORN_DOWN release them.

SELECT "organizationId",
       CASE WHEN "source" = 'legacy-grandfathered' THEN 'legacy' ELSE 'direct' END AS slot,
       count(*) AS connections
FROM "SsoConnection"
WHERE "state" NOT IN ('DISCARDED', 'TORN_DOWN')
GROUP BY "organizationId", slot
HAVING count(*) > 1;

SELECT domains.domain, count(DISTINCT connection."organizationId") AS organizations
FROM "SsoConnection" AS connection
CROSS JOIN LATERAL unnest(connection."verifiedDomains") AS domains(domain)
WHERE connection."state" NOT IN ('DISCARDED', 'TORN_DOWN')
GROUP BY domains.domain
HAVING count(DISTINCT connection."organizationId") > 1;

Resolve reported duplicates through the existing connection lifecycle before upgrading, or contact support. Do not delete the active connection or discard domain ownership to force the migration through.

When upgrading an installation that already has connection-scoped SCIM data, also check for directory identities whose connection is missing:

SELECT DISTINCT identity."connectionId"
FROM "ScimExternalId" AS identity
LEFT JOIN "SsoConnection" AS connection ON connection."id" = identity."connectionId"
WHERE connection."id" IS NULL;

If ScimDirectoryUser already exists, run the same query with that table as well. The ownership migration derives organizationId from the connection and refuses missing connections rather than guessing a tenant. It also backfills verified-domain routing for existing live connections of every source.

SCIM now keeps each organization’s directory profile and active state separately from the shared login account. A directory rename does not change the primary sign-in email, and deactivation removes access only to that organization. Inactive directory users remain available through SCIM reads. Group external identifiers are unique within each directory; legacy groups retain their organization-wide namespace.

Directory usernames remain unique within each organization, ignoring case and surrounding whitespace. Deleted resources release their usernames. The migration refuses conflicting existing usernames rather than merging accounts. If both directory identity tables already exist, this query must also return no rows:

WITH claims AS (
  SELECT "connectionId", "userId" FROM "ScimDirectoryUser"
  UNION
  SELECT "connectionId", "userId" FROM "ScimExternalId"
)
SELECT connection."organizationId", lower(btrim(account."email")) AS username,
       count(DISTINCT account."id") AS users
FROM claims
JOIN "SsoConnection" AS connection ON connection."id" = claims."connectionId"
JOIN "User" AS account ON account."id" = claims."userId"
WHERE btrim(account."email") <> ''
GROUP BY connection."organizationId", lower(btrim(account."email"))
HAVING count(DISTINCT account."id") > 1;

For an earlier schema containing only ScimExternalId, use only that table’s SELECT in the claims expression. Resolve any conflicts before upgrading.

The migration preserves existing global account disables. It cannot distinguish an earlier SCIM disable from an administrator’s security action, so it does not automatically reactivate those accounts. Review any historical disable through the administrator account controls when restoring access.

Docker Compose

# Pull latest images
docker compose pull

# Restart with new images
docker compose up -d

The workers run the upgrade when they start, and the app does not report ready until it has completed (see Run the upgrade). Compose has no separate migrate service; if your compose file predates this, copy the current one from infra/compose.yml. The workers must be enabled.

Helm Chart

# Update the Helm repository
helm repo update

# Upgrade the release
helm upgrade langwatch langwatch/langwatch \
  --namespace langwatch \
  -f values-production.yaml \
  --wait --timeout 45m

The upgrade can run for up to 35 minutes on a large installation, in the pre-upgrade Job where one renders (see Run the upgrade) and otherwise in the new worker pods. Helm’s default hook timeout is 5 minutes, and a shorter --timeout fails the release part-way through a migration.

Pin a Specific Version

# List available versions
helm search repo langwatch/langwatch --versions

# Install a specific chart version
# x-release-please-start-version
helm upgrade langwatch langwatch/langwatch \
  --namespace langwatch \
  --version 3.20.1 \
  -f values-production.yaml
# x-release-please-end

Run the upgrade

The api never migrates the database itself. The workers run the upgrade when they start; until the installation is current the api reports not ready, and a process started on an installation that has not been upgraded to its release refuses to serve and logs which steps are outstanding:

langwatch-api (api) refuses to serve: the installation is behind this image: blocking steps not done: prisma:20261006180000_..., clickhouse:00042. Run `pnpm task upgrade` from this image, then start it again.

Starting the workers on the new image runs the upgrade for you. To run it yourself, use the new image, then start the app again.

Each way of running LangWatch runs the upgrade for you:

How you run LangWatch	Who runs the upgrade
Helm, helm upgrade with upgrades serialised (serializeUpgrades on and no app.dataplane)	the pre-upgrade Job, on the new image, before any pod rolls
Helm, helm upgrade otherwise	the new worker pods, on start; the new api pods are not ready until it is done
Helm, first install	the worker’s first boot, once, on the empty database
Helm with serialised upgrades and app.migrations.preRoll: false	you, before helm upgrade (below)
Docker Compose	the workers, on every docker compose up; the app is not ready until it is done
npx @langwatch/server	the workers the CLI starts; the CLI has no migration phase of its own

To run it yourself, from the new image with the app’s environment (DATABASE_URL, CLICKHOUSE_URL, REDIS_URL):

cd /app/apps/api && pnpm --silent run start:prepare:db

That runs pnpm task upgrade (Postgres, ClickHouse and LangWatchQL, under one lease, so a second run waits for the first). It is safe to run again: on a current installation it applies no step and leaves the schema as it is. To see what it would do first, run cd /app/apps/tasks && pnpm task upgrade plan; pnpm task upgrade status shows the ledger. Both take --json.

Exit codes and what to do

pnpm task upgrade exits 0 when it is done and otherwise one of two non-zero codes, and its last two log lines name the cause and the next action. A refused or failed run is recorded, so Ops > Upgrades and upgrade status show it.

Exit	Outcome code	Meaning	What to do
0	done	Every step the image needs is applied	Start or restart the app and the workers
1	failed_prisma_migration	A PostgreSQL migration failed and cannot be retried by the upgrade (written before re-runnable migrations became the rule, or not shipped in this image)	Run the prisma migrate resolve command the message names, then run the upgrade again
1	rerunnable_migration_failed	A re-runnable migration still failed on its last attempt	Fix what its error names (often a lock another session holds), then run the upgrade again. No resolve command is needed
1	prebuilt_index_differs	A pending migration creates an index if it does not exist, and an index of that name already exists with another definition or invalid	Drop or rename the index the message names (the upgrade does not drop it for you), then run the upgrade again. A waiting worker retries by itself
1	schema_failed	A PostgreSQL or ClickHouse target could not be applied, or the ledger schema could not be created	Check the named target is reachable (DATABASE_URL, CLICKHOUSE_URL) and that the DATABASE_URL user has CREATE on the database, then run the upgrade again
1	step_failed	A blocking data step failed	Read its error with pnpm task upgrade status, fix the cause, then run the upgrade again. The step resumes from its checkpoint
1	reconciler_failed	The ClickHouse TTL or LangWatchQL reconciler failed	Fix the store it names (CLICKHOUSE_URL), then run the upgrade again
1	lease_lost	This run lost the upgrade lease part-way (it stopped renewing it)	Run the upgrade again. It resumes from the ledger
1	failed	Any other failure	Read the error and upgrade status, fix the cause, then run it again
3	lease_not_acquired	Another upgrade holds the lease and did not finish within 10 minutes	Wait for it (upgrade status shows its run), then run the upgrade again. A dead holder’s lease expires by itself after 60 seconds

Every 1 is safe to retry: the ledger remembers what is done.

When a PostgreSQL migration fails

PostgreSQL applies a migration one statement at a time, so a statement cancelled by the 2 second lock_timeout leaves its earlier statements applied and a failed row in _prisma_migrations. Every migration written after 20261006180001_gateway_realtime_session_metering is written to survive a second run. For those, upgrade marks the failed row rolled back itself (prisma migrate resolve --rolled-back <migration>), waits and applies it again, up to 3 attempts with a 2 second then 4 second wait.

A failed row an earlier run left is cleared the same way before the schema is applied.

An older migration, or a failed one this image does not ship, stops the run with failed_prisma_migration. Check what it left behind, run prisma migrate resolve --rolled-back <migration> if none of it remains (or --applied <migration> if you finished it by hand) with the app’s DATABASE_URL, then run the upgrade again. No migration has rolled forward, so the previous version keeps serving.

Large tables: build indexes ahead of the upgrade

Two schema changes touch a whole large table while the upgrade runs.

• AuditLog unique index (170500_audit_log_idempotency_key): PostgreSQL builds it in place and holds a lock that blocks audit writes for the build. On a large table the build can outlast the 2 second lock_timeout and fail. Build the index yourself first, during a quiet period, with CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS using the index name the migration declares. The migration then finds it and moves on.
• ClickHouse trace_summaries index (00103_trace_summaries_updated_at_index): the index is added and then materialised over every existing row. The mutation runs in the background and competes with merges for IO, and later ALTER statements on that table queue behind it. Watch system.mutations and run the upgrade at a quiet time on a large table.

What the upgrade does

• In release order. From the release the ledger records to the release of the image, the upgrade plans each release’s PostgreSQL migrations, then its ClickHouse migrations on every ClickHouse target (the shared one and each private dataplane), then its blocking data steps. An upgrade by one release applies its schema in one pass, then its blocking steps, as before.

  An upgrade across several releases steps through them: it says so, then for each release it names the release, applies the PostgreSQL migrations up to and including it, says how many it applied, and runs that release’s blocking steps before it moves on, so a blocking step sees the PostgreSQL schema of its own release.

  While stepping, Prisma’s own output is shown only when a migration fails. ClickHouse migrations step the same way: each release applies them on every ClickHouse target up to its own last one.

  Migrations the image ships that no release lists yet are applied in one pass at the end. A ClickHouse target that fails is reported after the others have run, and fails the upgrade.

• Its own record first. The upgrade keeps its ledger, its lease and the list of running processes in their own PostgreSQL schema, <your schema>_upgrade_ledger (public_upgrade_ledger by default), which it creates before anything else. The database user in DATABASE_URL needs CREATE on the database for that; if it has none, create the schema beforehand with that user as its owner.

• From any version. Upgrade to the latest release from any version; the worker rolls through every step. There is no release to stop at first. Releases and support explains which releases get fixes.

• Short locks and retries. PostgreSQL statements run with a 2 second lock_timeout. A statement that waits longer fails instead of blocking your traffic. The upgrade makes up to 3 attempts at a failed schema apply (waiting 2 seconds, then 4), then exits; the previous version keeps serving, and running it again resumes where it stopped.

• One run at a time. The upgrade holds a lease in its ledger, renews it every 15 seconds and loses it after 60 seconds without a renewal. A second run waits up to 10 minutes for a live holder, saying so every 30 seconds, then exits 3.

• Background steps. Data copies that do not block the release run on the workers after the upgrade. A step that must not run while an older version still writes waits until every running app and worker is on a version that knows it.

ClickHouse is required

The app and the workers refuse to start without ClickHouse. An installation with no CLICKHOUSE_URL, or one where the upgrade never applied the ClickHouse migrations, stays refused by name until pnpm task upgrade has run against a reachable ClickHouse.

While it runs, and after

Every running app and worker records itself in the upgrade ledger’s serving roster as it starts, refreshes that record every 15 seconds, and removes it on a graceful stop. A record not refreshed for 60 seconds counts as gone. The upgrade uses these records to tell when every older version has stopped. A process that cannot refresh its own record for 60 seconds stops serving until a refresh succeeds again, so it never keeps writing unseen.

To follow an upgrade:

Where	What it shows
Ops > Upgrades (/ops/upgrades)	The installation’s state, each release’s steps with their status and a one-line description, every run with its phases and error. It refreshes while a run is in progress. Platform operators only (ADMIN_EMAILS).
pnpm task upgrade status	The same state and steps, in the terminal, from the app image (--json for scripts)
Settings, Checkup, langwatch doctor	The Postgres migrations and ClickHouse migrations rows: Fail for a pending or failed migration, with the fix; Not checked when no upgrade is recorded yet or the ledger cannot be read
npx @langwatch/server doctor	The same output as upgrade status, for a local install

The installation is in one of seven states: Up to date, Finishing in background (blocking steps done, background steps still running), Behind (this image declares steps the ledger has not run), Never upgraded (an empty ledger), Upgrading, Rolled back (an older version is running beside newer steps) and Needs attention (a step or run failed).

Reading the console

The upgrade logs as langwatch:tasks:upgrade. Every progress line carries four fields next to its message: phase, waitingOn (what it is blocked on, with a placeholder word when the run is not waiting), elapsedMs since the run began, and next (what you can do). A password, token or connection-string credential is masked in every line. These are the line families, in the order a run writes them:

Line starts with	When	What it tells you
first run: this database holds no LangWatch schema	The database is empty	The upgrade creates its ledger, applies every migration, then exits so the app can serve
no ClickHouse target is configured (warning)	CLICKHOUSE_URL and every private route are unset	The upgrade applies PostgreSQL only; the app and workers refuse to serve until you set it
upgrade ledger ready in Postgres schema <schema> after N ms	Start of every run	Where the ledger lives. the upgrade ledger could not be created (warning) ends the run with schema_failed
waiting for the upgrade lease held by <owner> on <host> (<image>)	Another run holds the lease	Who holds it and how long you have waited, every 30 seconds
upgrade planned from <release>: N schema migrations (a Postgres, b ClickHouse) and N blocking steps across <releases>	After the plan	What this run will do. A line that says the ledger is current means it will do none
phase <name> started: waiting on ..., then succeeded in N ms or failed after N ms (warning)	Each phase	The phases are preflight, postgres-schema, clickhouse-schema (per release) and reconcile
blocking step <id> started (from the start | resuming from its checkpoint): <description>, then done in N ms or failed after N ms: <error>	Each blocking data step	The step’s id and its one-line description
Prisma migration <name> failed ... and is re-runnable: marked it rolled back (warning), then applied on attempt N	A migration cancelled by a lock	Which migration, which attempt of 3, and the wait before the next. schema apply failed without a failed migration; retrying in N ms is the same retry for a transient failure
upgrade finished in N ms or first run finished in N ms; or upgrade stopped after N ms (<outcome code>) (warning)	End of the run	The outcome code from the table above
upgrade complete; the UI is at <BASE_HOST> ... or upgrade exited <code>; ...	Last line	Where the UI is, and that pnpm task upgrade status shows every step and run. With BASE_HOST unset it says so

A serving app or worker writes its own lines before it serves:

Line	Meaning
<app> (<role>) refuses to serve: the installation is behind this image: blocking steps not done: <ids>. Run pnpm task upgrade ...	The upgrade has not run for this image
... refuses to serve: ClickHouse is required and no ClickHouse target is configured	Set CLICKHOUSE_URL, run the upgrade, start again
first install: the ledger and the schema are empty, so this api runs pnpm task upgrade once before it serves	The Helm first-install path; the upgrade’s lines follow
roster entry lapsed past 60000 ms: <process> stops serving; readiness answers 503 until a roster write succeeds	The process cannot write to PostgreSQL. Check DATABASE_URL. roster entry written again: <process> serves again reports the recovery

A serving process that cannot read the ledger names DATABASE_URL in its refusal.

Upgrade settings

Variable or value	Effect
app.migrations.preRoll (Helm, default true)	Renders the pre-upgrade Job, only when upgrades are serialised (serializeUpgrades on and no app.dataplane); otherwise the new worker pods run the upgrade. false with serialised upgrades leaves it to you; until you run it, the new api is not ready.
app.migrations.preRollTimeoutSeconds (Helm, default 2100)	How long that Job may run.
CLICKHOUSE_MIGRATE_WAIT_SECONDS (default 180)	How long the upgrade waits for ClickHouse to accept connections. 0 fails on the first refused connection. A value that is not a number of seconds, 0 or more, is refused with Invalid CLICKHOUSE_MIGRATE_WAIT_SECONDS.
SKIP_LWQL_PROVISION (1 or true)	The upgrade skips LangWatchQL provisioning.
CLICKHOUSE_CLUSTER	Names the ClickHouse cluster. When set, the upgrade creates Replicated engines and replicates DDL across nodes.
CLICKHOUSE_COLD_STORAGE_ENABLED (true)	The reconcile phase writes table TTLs that move aged data to the cold volume. Hot days come from CLICKHOUSE_COLD_STORAGE_<TABLE>_TTL_DAYS, else CLICKHOUSE_COLD_STORAGE_DEFAULT_TTL_DAYS, else the table’s built-in default.
BASE_HOST	Only the upgrade’s last log line reads it, to say where the UI is.
BUILD_TIME	Set while building an image: the upgrade finds no ClickHouse target and applies no migration there. Leave it unset at runtime.

For the split into api, worker and tasks applications, see the chart’s upgrade notes. app.features.skipEnvValidation is inert: no process reads SKIP_ENV_VALIDATION any more.

Rollback

A rollback never runs migrations and never needs a down migration. Every migration in the supported window only adds, so the previous version runs on the upgraded schema as it is. The pre-upgrade Job, where one renders, runs on helm upgrade only, never on helm rollback.

• The previous version starts on the newer schema.
• The Upgrades page shows Rolled back while the older version runs. Background steps that an older version can undo are reopened, and run again after you upgrade again.
• Data the newer version wrote stays. Features the older version does not know about are unavailable until you upgrade again.

Helm

# List release history
helm history langwatch --namespace langwatch

# Rollback to a previous revision
helm rollback langwatch <revision> --namespace langwatch

Docker Compose

Pin image tags to a specific version instead of latest:

services:
  app:
    image: langwatch/langwatch:<previous-version> # the release you are rolling back to

Then restart:

docker compose up -d

Breaking Changes

Internal authentication for the NLP service

The app can now authenticate itself to the NLP service with a shared credential, LANGWATCH_NLP_INTERNAL_SECRET. The app sends it on every NLP request, and the NLP service refuses requests that do not carry it. With the variable unset on the NLP service, it accepts unauthenticated requests as before.

An upgrade requires no change, and an install that never sets the variable keeps working. Setting it is recommended, and the value has to be the same on both sides:

• Docker Compose: the app, the workers and langwatch_nlp all read .env, so add one LANGWATCH_NLP_INTERNAL_SECRET line there (openssl rand -hex 32) and restart the stack.

• Helm chart: with autogen.enabled=true the chart generates the value into the app Secret and points the app, the workers and the NLP Deployment at the same key, so a default install needs no change. A value already in that Secret is kept, so an upgrade does not rotate it.

  If you supply the app Secret yourself, add a LANGWATCH_NLP_INTERNAL_SECRET key to it and roll the three Deployments. Until you do, the key is absent, the variable stays unset on all three, and the install carries on.

Docker Compose also no longer publishes the NLP service on host port 5561. The app and the workers reach it over the compose network at langwatch_nlp:5561, which is what LANGWATCH_NLP_SERVICE already points at. If you were calling port 5561 from the host, add a ports entry for langwatch_nlp in your own compose override.

LangWatch AI Gateway Ingress moved to the app Ingress (Helm)

The gateway sub-chart used to create its own Ingress on every install, with class nginx, host gateway.langwatch.ai, TLS secret gateway-tls, a letsencrypt-prod cert-manager issuer and ingress-nginx streaming annotations. It now renders no Ingress. The gateway.ingress.* values are gone.

To keep publishing the gateway through an Ingress, set its host on the app Ingress. The chart renders a second Ingress, <release>-gateway-ingress, with the app Ingress className, labels and annotations:

ingress:
  enabled: true
  className: nginx
  # ...your existing app hosts, tls and annotations
  gateway:
    host: gateway.acme.com
    tls:
      secretName: gateway-tls

With className: nginx the chart still adds the streaming and timeout annotations to the gateway Ingress, and now sets a 32 MB body limit. The paths are fixed at /v1 (Prefix) and /health (Exact). For other paths, such as /v1beta, write your own route. If you relied on a cert-manager issuer only on the gateway Ingress, add it to ingress.annotations or ingress.gateway.annotations.

A values file that still sets gateway.ingress.* fails to render with a message naming ingress.gateway.host, so the Ingress cannot disappear on upgrade without an error. gateway.ingress.enabled: false on its own is accepted. If you route to the gateway another way, remove the block and set gateway.publicUrl. See Routing and ingress controllers.

The gateway PodDisruptionBudget now uses maxUnavailable: 1 and renders only when the gateway runs more than one pod. The app, workers, NLP and LangEvals budgets also render only above one replica. The chart refuses a budget that leaves no pod evictable, such as minAvailable equal to replicaCount or maxUnavailable: 0.

Helm installs with autogen off and Argo CD

With autogen.enabled=false, chart-managed Redis and PostgreSQL now need redis.auth.existingSecret and postgresql.auth.existingSecret (or a password value). The chart used to generate these passwords, and under Argo CD a generated password changed on every sync. A plain helm upgrade of a release that already has the chart’s own <release>-redis and <release>-postgresql Secrets keeps them.

Under Argo CD, first copy the password PostgreSQL currently accepts into a Secret of your own with another name: Argo CD prunes the chart’s Secret once existingSecret is set.

The LangWatchQL render Job is now named <release>-lwql-access-render-<hash> from its spec, and the install Job no longer expires. Argo CD no longer recreates it every few minutes or fails the sync on an immutable Job. Missing LangWatchQL passwords no longer block ClickHouse: the Job logs the missing keys and LangWatchQL stays off.

Keep Argo CD pruning on, so a new spec replaces the old Job; without it, delete old render Jobs before a rollback, or the rollback finds its Job already complete and does not render again.

The app now reads the gateway keys named by gateway.secrets.internalSecretKey and gateway.secrets.jwtSecretKey, the same keys the gateway pod reads.

3.16 or 3.17 to 3.18: Sign-in and SSO

The upgrade keeps existing accounts working with no manual step, apart from the cases below. A Helm upgrade from 3.16.0 with email and password users, an Okta-configured deployment and domain SSO was tested end to end; the Azure AD account move is covered by tests only:

• Password users sign in with their existing password on the new email-first sign-in page.
• SSO users land on their existing user and organization. The callback URL stays /api/auth/callback/<provider>, so the identity provider registration does not change.
• Azure AD accounts move to the key the new sign-in looks them up by on each user’s first sign-in after the upgrade. There is no SQL to run.
• Browser sessions and API keys keep working.
• The identity migrations run on first start. Check Ops > Upgrades > Tenant migrations afterwards; see Check the migration results.

The first start runs a migration preflight before the app reports ready. --timeout 45m on helm upgrade gives it room to start. A migration that cannot finish during the preflight stays gated and does not block startup, so check Ops > Upgrades > Tenant migrations once the rollout completes. The app can restart once or twice while the bundled ClickHouse restarts on its new image; that is expected.

Provider setting. The chart now passes app.nextAuth.provider to the app as AUTH_PROVIDER. NEXTAUTH_PROVIDER is still read and logs a deprecation warning. If you select the provider with NEXTAUTH_PROVIDER in app.extraEnvs, the chart leaves the setting to you, as before. Outside the chart, set AUTH_PROVIDER; when both are set it wins.

Moving password users to SSO. An SSO sign-in joins an existing password account only once its address is confirmed. See Existing email and password accounts for how linking works in the current release.

Okta and Auth0. The id token is now verified: its iss must equal the issuer in the provider’s discovery document. A provider behind a proxy that rewrites one but not the other stops signing people in until the two agree.

Dataset Object Storage (Helm)

Datasets moved from PostgreSQL to object storage, with a one-time automatic migration on upgrade. Helm deployments running more than one worker replica with the default local storage must either scale workers to 1 or configure S3-compatible storage before upgrading.

See the dataset storage migration guide for the upgrade checklist, storage options, and rollback notes.

v1.x, v2.x to v3: ClickHouse Migration

LangWatch v3 replaces Elasticsearch/OpenSearch with ClickHouse as the primary data store. This is a zero-downtime migration, Elasticsearch and ClickHouse run side-by-side while you migrate historical data.

See the full migration guide for step-by-step instructions covering infrastructure setup, data migration, and cleanup.

Helm Chart Pre-1.0.0 to 1.0.0+

If upgrading from a Helm chart version before 1.0.0, the PostgreSQL PVC naming changed. To preserve your data:

postgresql:
  primary:
    persistence:
      existingClaim: "data-langwatch-postgresql-0" # Your existing PVC name

Check your existing PVC name:

kubectl -n langwatch get pvc

Known Issues

3.20.1 drops OTLP/JSON spans with no attributes

LangWatch 3.20.1 drops an OTLP/JSON span that has no attributes array, but still answers 2xx, so the SDK reports success. The api logs Error parsing OTLP span, dropping once per lost span. OTLP allows a span with no attributes, and later releases accept it. Spans sent to 3.20.1 this way are not recovered by upgrading; upgrade to stop the loss.

Version Compatibility

Chart Version	App Version	Kubernetes	Helm	Data Store
3.x	3.x	1.28+	3.12+	ClickHouse
2.x	2.x	1.25+	3.10+	Elasticsearch, OpenSearch
1.x	1.x	1.25+	3.10+	Elasticsearch, OpenSearch

Getting Help

If you encounter issues during an upgrade:

• Check pod logs: kubectl -n langwatch logs deploy/langwatch-app --tail=100
• Check the Troubleshooting guide
• Open an issue at github.com/langwatch/langwatch/issues
• Contact support