> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.rightbrain.ai/v-1/docs/production/chaos-testing/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.rightbrain.ai/_mcp/server.

# Chaos testing

> Configure run-scoped tool errors, tool latency, and primary-model failures; reproduce them with a seed; and inspect the resulting chaos events.

Chaos testing lets you exercise an agent's failure handling without changing its revision or waiting for a provider incident. The policy applies to one agent run and can inject tool failures, malformed tool results, latency, or a primary-model failure.

> **Warning**
>
> Run chaos tests against test agents and test data. A chaos policy can skip a real tool execution, delay a run, or force the primary model to fail.

## Run-scoped policy

Send the policy in the `chaos` field when you call `POST .../task-agent/{agent_id}/run`. JSON requests accept a `TaskAgentChaosConfig` object. Multipart requests accept the same object as a JSON-encoded string.

| Field        | Default          | Contract                                                             |
| ------------ | ---------------- | -------------------------------------------------------------------- |
| `enabled`    | `true`           | When `false`, the run proceeds without a chaos policy.               |
| `seed`       | Server-generated | Up to 128 characters. Persisted on the run for replay.               |
| `run_index`  | `0`              | Integer from 0 through 100,000 used to vary deterministic decisions. |
| `max_faults` | `1`              | Integer from 1 through 10.                                           |
| `modes`      | —                | One through 10 modes when chaos is enabled.                          |

The policy is not stored on the agent revision. Normal runs remain unaffected.

## Fault modes

Each mode has a `probability` from 0 through 1 and a `type`.

| Type                        | Required fields                     | Behavior                                                                                                                                                                                         |
| --------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tool_error`                | `type`, `probability`               | Skips an eligible tool call and returns a synthetic tool error. Set `retryable` to mark the error as retryable.                                                                                  |
| `tool_error` with `variant` | `type`, `probability`, `variant`    | Replaces the tool result with `empty_result`, `missing_fields`, `null_fields`, `unexpected_nesting`, `truncated_result`, or `stale_result`. A variant cannot be combined with `retryable: true`. |
| `tool_latency`              | `type`, `probability`, `latency_ms` | Adds deterministic latency before an eligible tool call. `latency_ms.min` and `latency_ms.max` must each be between 0 and 5,000.                                                                 |
| `model_error`               | `type`, `probability`, `error`      | Fails a primary-model call with `timeout`, `rate_limit`, `provider_unavailable`, or `transient_internal`.                                                                                        |

Tool modes can include `targets.tool_names` and `targets.sources`. Supported sources are `task`, `mcp`, `integration`, and `registered_tool`. Omit `targets` to make every tool call eligible for that mode.

## Deterministic replay

Each decision is derived from the seed, agent revision, request fingerprint, `run_index`, execution phase, call position, target, and mode position. Reusing the same policy and request against the same revision reproduces the same sampling decisions. Change `run_index` to exercise a different deterministic run without changing the seed.

If you omit `seed`, Rightbrain generates one and returns it in `chaos_config`. Use that stored value for a replay. Approval resumes preserve the run's existing policy, applied-fault count, latency budget, and event history.

## Safety bounds

`max_faults` limits how many sampled faults can be applied. Tool latency is capped at 5 seconds per fault and 15 seconds across the run. Decisions that would exceed the fault or latency budget are retained as unapplied events with a reason.

These bounds prevent an unbounded policy, but they do not make a chaos run suitable for production traffic.

## Inspect a chaos run

Agent run records expose:

* `is_chaos_run` — whether the run has an enabled policy.
* `chaos_config` — the normalized policy, including its replayable seed.
* `chaos_events` — every eligible sampling decision in execution order.

Each event records whether the fault was applied, its probability threshold and sampled value, the target and phase, and any mode-specific error, variant, retryability, or latency. Decision keys are returned as hashes.

Use `chaos_enabled` when listing agent runs to select chaos or non-chaos history.

## Fallbacks and approvals

A `provider_unavailable` or other model fault exercises the same primary-model failure path as a provider incident. If the agent revision has an eligible fallback model, the run records the failover in its normal model-usage fields.

Tool faults continue through the standard tool-result pipeline. Approval-gated tools still follow their configured approval policy, and a resumed run continues with the original chaos budget and event record.

## Related

#### [Fallbacks & reliability](/docs/production/fallbacks)

Configure the model failover path that a chaos run can exercise.

#### [Observability & audit](/docs/production/observability)

Read run identity, events, model usage, and failure details.

#### [Run agents via the API](/docs/api/run-agents)

Send run requests and consume the SSE event stream.

#### [API Reference: Agents](/api-reference/api-reference/agents)

Read the complete run and chaos schemas.

> Build reliable AI agents that run inside your existing tools and workflows. Rightbrain developer documentation.