Backups

Backup and restore strategies for LangWatch data stores

LangWatch stores data across three systems. Each requires its own backup strategy:

Data Store	What It Stores	Backup Priority
PostgreSQL	Users, teams, projects, configurations, prompt versions	Critical
ClickHouse	Traces, spans, evaluations, experiments, analytics	High
S3	Datasets, ClickHouse cold data	Medium

PostgreSQL Backups

PostgreSQL holds your control plane data, losing it means losing user accounts, project configurations, and monitor definitions.

Chart-Managed PostgreSQL

If you’re using the chart-managed PostgreSQL (development/small deployments), use pg_dump:

# Create a backup
kubectl exec -n langwatch statefulset/langwatch-postgresql -- \
  pg_dump -U postgres langwatch > backup-$(date +%Y%m%d).sql

# Restore from backup
kubectl exec -i -n langwatch statefulset/langwatch-postgresql -- \
  psql -U postgres langwatch < backup-20260407.sql

External PostgreSQL (RDS, Cloud SQL, etc.)

For production, use your cloud provider’s built-in backup features:

• AWS RDS: Enable automated snapshots (recommended: 30-day retention) and point-in-time recovery
• GCP Cloud SQL: Enable automated backups with point-in-time recovery
• Azure Database: Enable geo-redundant backups

Always test your restore procedure before you need it. Schedule a quarterly restore drill to validate your backups.

ClickHouse Backups

ClickHouse holds all your trace and evaluation data. The clickhouse-serverless subchart supports native ClickHouse BACKUP/RESTORE to S3-compatible storage.

Enable Backups

Backups require an S3-compatible bucket. Configure in your Helm values:

clickhouse:
  # S3 bucket for backups (shared with cold storage if both enabled)
  objectStorage:
    bucket: "my-langwatch-backups"
    region: "us-east-1"
    useEnvironmentCredentials: true # IRSA / workload identity

  backup:
    enabled: true
    database: "langwatch"
    user: "default"
    full:
      schedule: "0 */12 * * *" # Full backup every 12 hours
    incremental:
      schedule: "0 * * * *" # Incremental every hour

Or use the cold-storage-s3.yaml overlay which enables both cold storage and backups:

helm install langwatch langwatch/langwatch \
  -f examples/overlays/size-prod.yaml \
  -f examples/overlays/cold-storage-s3.yaml \
  --set clickhouse.objectStorage.bucket=my-langwatch-backups \
  --set clickhouse.objectStorage.region=us-east-1

The overlay leaves bucket and region empty, and the chart refuses to render without them.

S3 Authentication

IRSA, Workload Identity (recommended):

clickhouse:
  objectStorage:
    useEnvironmentCredentials: true
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/clickhouse-s3-role

Static credentials:

clickhouse:
  objectStorage:
    useEnvironmentCredentials: false
    credentials:
      secretKeyRef:
        name: clickhouse-s3-creds # K8s secret name
        accessKeyId: "accessKey" # key in the secret that holds the access key ID
        secretAccessKey: "secretKey" # key in the secret that holds the secret access key

Backup Schedule

Backup Type	Default Schedule	Description
Full	0 */12 * * * (every 12h)	Complete database backup
Incremental	0 * * * * (every 1h)	Only changes since last full backup

Both run as Kubernetes CronJobs (<release>-clickhouse-backup-full and <release>-clickhouse-backup-incremental). Each job starts its own pod from the ClickHouse image and runs the backup against the first ClickHouse replica over the network.

Restore from Backup

To restore, you need to identify the backup name and run the restore command:

# List available backups
kubectl exec -n langwatch langwatch-clickhouse-0 -- \
  clickhouse-client --query "SELECT * FROM system.backups ORDER BY start_time DESC"

# Restore a specific backup
kubectl exec -n langwatch langwatch-clickhouse-0 -- \
  clickhouse-client --query "RESTORE DATABASE langwatch FROM S3('https://s3.us-east-1.amazonaws.com/my-langwatch-backups/backup-name', 'access_key', 'secret_key')"

Restoring a backup overwrites existing data in the target database. Always verify you’re restoring to the correct environment.

ClickHouse Cold Storage

Cold storage is separate from backups, it’s a tiered storage strategy that automatically moves older data from local SSD to S3 for cost savings.

How It Works

1. New data is written to hot storage (local SSD on the ClickHouse pod)
2. After the TTL period, data is moved to cold storage (S3)
3. Queries transparently read from both hot and cold storage
4. Cold data is cached locally for repeated reads

Enable Cold Storage

clickhouse:
  objectStorage:
    bucket: "my-langwatch-data"
    region: "us-east-1"
    useEnvironmentCredentials: true

  cold:
    enabled: true
    defaultTtlDays: 49 # Data older than 49 days moves to S3

We recommend setting the TTL to a multiple of 7 (e.g., 7, 14, 28, 49) to align with ClickHouse’s weekly partition boundaries for more efficient data management. The default of 49 days means data stays on fast local storage for ~7 weeks before moving to S3.

Cost Savings

Cold storage can reduce storage costs significantly:

Storage Type	Approximate Cost	Speed
gp3 SSD (hot)	~$0.08/GB/month	Fast
S3 Standard (cold)	~$0.023/GB/month	Slower (cached)
S3 Infrequent Access	~$0.0125/GB/month	Slower

For a deployment with 150 GB/month of trace data, cold storage can save ~$500/year.

S3 Dataset Backups

If you’re using S3 for dataset storage (app.dataplane.enabled: true), protect this data with:

• S3 Versioning: Enable versioning on the bucket to recover from accidental deletes
• Cross-region replication: For disaster recovery, replicate to another region
• Lifecycle policies: Move old versions to Glacier after 30 days

Disaster Recovery Checklist

• PostgreSQL automated backups enabled (30-day retention)
• ClickHouse backup CronJobs running (check kubectl get cronjobs)
• S3 bucket versioning enabled
• Backup S3 bucket is in a different region or account from primary
• Restore procedure documented and tested
• Quarterly restore drills scheduled