Dataset Storage Migration

Upgrade guide for Helm deployments: datasets move to object storage, with a one-time migration you run once

Datasets are now stored in object storage instead of PostgreSQL. This makes large datasets faster to load, cheaper to store, and enables direct browser uploads. New datasets go to the new storage backend as soon as you upgrade. Existing datasets keep working from PostgreSQL until you run the one-time migration that moves them.

The migration never deletes your data. All dataset rows are kept in PostgreSQL as a fallback, so you can roll back safely. It is also safe to re-run: if it is interrupted or a dataset fails to migrate, the next run picks up where it left off.

​

What Changed

• Dataset storage: dataset content is written to object storage (a persistent volume by default, or S3-compatible storage if configured)
• Migration step: a background upgrade step moves existing datasets. The workers run it on their own once every old-version pod is gone, see Running the migration
• Local storage constraints: with the default local storage, the app and workers now restart together during upgrades, and workers are limited to a single replica
• New chart values: see the chart README for the full app.dataplane and app.storedObjects parameter reference

​

Before You Upgrade

Pick the path that matches your deployment. If you run more than one worker replica, this choice is required, not optional: see Multiple worker replicas below.

​

Path A: Default local storage

If you have not configured S3-compatible storage, datasets are stored on the same persistent volume the chart already uses for attachments and file uploads. No configuration is needed, but be aware of three things:

1. Check volume headroom. The migration writes your existing datasets to the stored-objects volume (10Gi by default, shared with attachments and staged uploads). Before upgrading, compare your dataset sizes in PostgreSQL against the free space on that volume. To estimate dataset size:

SELECT pg_size_pretty(pg_total_relation_size('"DatasetRecord"'));

If you need more space, grow the volume first:

app:
  storedObjects:
    localFilesystem:
      size: 50Gi

2. Expect a brief restart during upgrades. With local storage, the app and workers restart together on each upgrade (the volume can only be attached to one node at a time). This means a short window of downtime per upgrade. If you need zero-downtime upgrades, use Path B.

3. Pods are scheduled together. The workers share the app’s volume, so they are placed on the same node as the app pod automatically.

​

Path B: S3-compatible object storage

For AWS S3, MinIO, or any S3-compatible service, enable the dataset storage backend:

app:
  dataplane:
    enabled: true
    bucket: "your-dataset-bucket"
    providers:
      awsS3:
        accessKeyId:
          secretKeyRef: { name: "your-secret", key: "accessKeyId" }
        secretAccessKey:
          secretKeyRef: { name: "your-secret", key: "secretAccessKey" }
        # For MinIO or other S3-compatible services:
        # endpoint:
        #   value: "https://minio.internal:9000"
  storedObjects:
    localFilesystem:
      enabled: false

With object storage enabled, none of the local-storage constraints apply: no persistent volume, no single-node scheduling, no restart window, and no worker replica limit.

Two requirements for the bucket:

1. CORS: the browser uploads dataset files directly to the bucket, so your CORS policy must allow PUT requests from your LangWatch app origin:

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://langwatch.your-domain.com"],
      "AllowedMethods": ["PUT"],
      "AllowedHeaders": ["*"]
    }
  ]
}

2. Lifecycle rule: in-progress uploads are staged under the staging/ prefix before being committed. Add a lifecycle rule that expires objects under staging/ after 1 to 7 days so abandoned uploads do not accumulate.

On EKS with IRSA, put the role on the ServiceAccount the app and workers use (global.serviceAccount). The migration runs inside a workers pod, so it uses the same identity.

​

Multiple Worker Replicas Require Object Storage

With local storage active, the chart now refuses to install or upgrade when workers.replicaCount is greater than 1:

app.storedObjects.localFilesystem.enabled requires workers.replicaCount=1
... Enable app.dataplane for multi-replica deployments.

This is deliberate: the local volume can only be safely written from one node. Before upgrading, either:

• set workers.replicaCount: 1, or
• configure object storage (Path B)

​

Running the Upgrade

Use --wait when crossing this version boundary, so the new pods are serving before you migrate:

helm upgrade langwatch langwatch/langwatch \
  --namespace langwatch \
  -f values-production.yaml \
  --wait --timeout 15m

New datasets created after the upgrade go straight to the new storage backend. Existing datasets keep working from PostgreSQL until you migrate them.

​

Running the Migration

You do not start the migration. Once every old-version pod is gone, the workers run the dataset:move-content-to-object-storage upgrade step in the background. Run the upgrade off-peak if you can. Its progress and report show on Ops > Upgrades.

The migration is resilient by design:

• Resumes where it stopped: an interrupted or partial run carries on from its last checkpoint
• Non-destructive: PostgreSQL rows are kept as a fallback and are not deleted
• Detects most concurrent edits: a dataset whose records change while it is being copied is skipped, and the step stays unfinished. The check is best-effort: an edit in the moment between the final check and the switch can be missed, so avoid dataset edits while the step runs
• Per-dataset failures do not stop the run: a dataset that fails is logged and stays on PostgreSQL, readable as before

The step’s report lists how many datasets moved, were skipped or failed.

​

Rolling Back

Rolling back to an older image after the migration is safe for reads: all dataset rows are still in PostgreSQL, so the old version serves them normally.

One caveat: dataset edits made while rolled back are written only to PostgreSQL. After you re-upgrade, those edits do not appear in datasets that were already migrated, and reconciling them requires manual work. Keep rollback windows short and avoid dataset edits during them.

​

Troubleshooting

​

Pods stuck Pending on bare-metal clusters

If your cluster has no default StorageClass, the stored-objects volume claim stays Pending, pods queue behind it, and helm upgrade fails at its --timeout. Either install a dynamic provisioner (for example local-path-provisioner) or point the chart at an existing StorageClass:

app:
  storedObjects:
    localFilesystem:
      storageClassName: "your-storage-class"

​

Migration failed

Anything left unmigrated stays readable from PostgreSQL. The step fails and names the dataset and the error. Fix the cause, then press Retry step on Ops > Upgrades; it resumes from its last checkpoint.

​

Getting Help

• Full parameter reference: chart README
• Check the Troubleshooting guide
• Open an issue at github.com/langwatch/langwatch/issues