> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details.

# Operations Console

> Run your LangWatch installation: inspect queues, recover processing and support identity or directory issues.

The **Ops Console** (`/ops`) is a platform-wide operations interface built for teams running LangWatch on their own infrastructure. It gives operators real-time visibility into the event-sourcing pipeline, queue health, processing throughput, error clusters, and the tools to act on problems without touching the database or restarting pods.

![Ops Dashboard overview](https://mintcdn.com/langwatch/Bcv1mdi2Gl7gtdK5/images/ops/dashboard.png?fit=max\&auto=format\&n=Bcv1mdi2Gl7gtdK5\&q=85\&s=7a47f7ddda9798675b2652c7c2d6a981)

## When to Use Ops

| Scenario | Where to go |
| - | - |
| Traces aren't appearing in the UI | [Dashboard](https://langwatch.ai/docs/self-hosting/ops/dashboard.md): check Staged/s and Failed/s rates |
| Processing is stuck | [Queue Management](https://langwatch.ai/docs/self-hosting/ops/queue-management.md): inspect blocked groups and error clusters |
| A deployment broke a projection | [Projection Replay](https://langwatch.ai/docs/self-hosting/ops/projection-replay.md): replay affected projections from a known-good date |
| Need to debug a specific trace or aggregate | [Deja View](https://langwatch.ai/docs/self-hosting/ops/dejaview.md): time-travel through the event stream |
| Want to test trace ingestion against a project | [The Foundry](https://langwatch.ai/docs/self-hosting/ops/foundry.md): build and send synthetic traces |
| SSO setup or directory sync needs investigation | The identity and directory Backoffice pages below |
| An upgrade is running, behind or failed | `/ops/upgrades`: the installation's state, each release's steps and every run ([Upgrade Guide](https://langwatch.ai/docs/self-hosting/upgrade.md#while-it-runs-and-after)) |
| An upgrade is waiting on an organization migration | `/ops/upgrades?tab=tenants`: Ops > Upgrades > Tenant migrations, to inspect its result and report |
| Failed jobs need reprocessing | [Queue Management](https://langwatch.ai/docs/self-hosting/ops/queue-management.md): redrive from the DLQ |

## Feature Overview

> [card] [Dashboard](https://langwatch.ai/docs/self-hosting/ops/dashboard.md)
>
> Real-time throughput, latency, error rates, and pipeline health at a glance. Powered by
> Server-Sent Events with automatic polling fallback.

> [card] [Queue Management](https://langwatch.ai/docs/self-hosting/ops/queue-management.md)
>
> Error groups, blocked queues, dead letter queue redriving, draining, and pipeline pause/unpause
> controls.

> [card] [Projection Replay](https://langwatch.ai/docs/self-hosting/ops/projection-replay.md)
>
> Rebuild projection state by replaying events from ClickHouse. Supports bulk replay across
> tenants, single-aggregate debugging, and dry runs.

> [card] [Deja View](https://langwatch.ai/docs/self-hosting/ops/dejaview.md)
>
> Time-travel debugger for event-sourced aggregates. Inspect the full event history and compute
> any projection's state at any point in time.

> [card] [The Foundry](https://langwatch.ai/docs/self-hosting/ops/foundry.md)
>
> Interactive trace playground for building, visualizing, and sending synthetic traces to any
> project. Useful for testing ingestion pipelines and reproducing issues.

## Who can use Ops

Ops access comes from the deployment's `ADMIN_EMAILS` allow-list. Set it to a
comma-separated list of trusted operator email addresses:

```bash theme={null}
ADMIN_EMAILS=operator@yourcompany.com,backup-operator@yourcompany.com
```

These operators have platform-wide access. Being an organization Admin, or
creating a custom role with `ops:view` or `ops:manage`, does not grant Ops
access. The console uses those permission names to distinguish reads and
changes, but the current allow-list grants the operator both.

The Backoffice pages also check the platform-admin allow-list. Use
[organization roles](https://langwatch.ai/docs/platform/rbac.md) for customer access to teams, projects and
organization settings.

## Identity and directory support

Organization administrators manage their own provider in **Settings >
Authentication**, and people and assignments in **Settings > Users**.
Use the Backoffice pages when an installation operator needs to investigate
across organizations.

| Page in your installation | Use it for |
| - | - |
| `/ops/backoffice/identity-lookup` | Find an account, inspect its identity state and sign it out everywhere when required. |
| `/ops/backoffice/sso-connections` | Inspect connection state and domain evidence, investigate setup, or pause and resume a connection. |
| `/ops/backoffice/directory-sync` | Inspect SCIM sync failures, provisioning attempts and external user mappings. Retry a retired apply after fixing its cause. |
| `/ops/upgrades` | Follow an upgrade: the installation's state, each release's steps and every run with its phases and error. |
| `/ops/upgrades?tab=tenants` | Ops > Upgrades > Tenant migrations: inspect migration progress and reports, or run a targeted migration for an organization. |

Pausing SSO stops new sign-ins through the connection. It leaves existing
sessions and SCIM provisioning in place. Removing a connection also stops
SCIM requests immediately and later revokes that connection's tokens. It does
not remove members or end their sessions. To offboard a person, use Directory
or the identity provider's SCIM deactivation before removing the connection.
See [SSO removal](https://langwatch.ai/docs/platform/sso.md#disable-or-remove-a-connection).

For a migration, read the per-organization result. **Held** means the work ran
but its parity checks found differences. **Parked** means the attempt failed
and will be retried. **Finalized** means the organization completed that
migration. A run that is waiting for an earlier migration has not changed the
organization yet. Do not treat starting a migration as proof it finished.

Identity and access migrations run automatically after each upgrade; see
[Automatic identity and access migration](https://langwatch.ai/docs/self-hosting/upgrade.md#automatic-identity-and-access-migration).
Check every required migration, including users as well as organizations.
Credential repair is recurring and does not finalize; the one-time backfills
must finalize.

## Architecture Context

The Ops Console sits on top of LangWatch's event-sourcing pipeline:

```mermaid theme={null}
graph LR
    Ingest["Trace Ingestion"] -->|"commands"| Redis["Redis Queues"]
    Redis -->|"consume"| Workers["Workers"]
    Workers -->|"events"| CH["ClickHouse"]
    Workers -->|"projections"| CH
    Ops["Ops Console"] -.->|"reads"| Redis
    Ops -.->|"reads"| CH
    Ops -.->|"actions"| Redis
```

* **Commands** enter Redis queues from the ingestion API
* **Workers** consume commands, emit events, and update projections in ClickHouse
* The **Ops Console** reads queue state from Redis and event history from ClickHouse, and can issue control actions (unblock, drain, pause, replay) back to the queues


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.