Chaos testing

Chaos testing lets you exercise an agent’s failure handling without changing its revision or waiting for a provider incident. The policy applies to one agent run and can inject tool failures, malformed tool results, latency, or a primary-model failure.

Run-scoped policy

Send the policy in the chaos field when you call POST .../task-agent/{agent_id}/run. JSON requests accept a TaskAgentChaosConfig object. Multipart requests accept the same object as a JSON-encoded string.

The policy is not stored on the agent revision. Normal runs remain unaffected.

Fault modes

Each mode has a probability from 0 through 1 and a type.

Tool modes can include targets.tool_names and targets.sources. Supported sources are task, mcp, integration, and registered_tool. Omit targets to make every tool call eligible for that mode.

Deterministic replay

Each decision is derived from the seed, agent revision, request fingerprint, run_index, execution phase, call position, target, and mode position. Reusing the same policy and request against the same revision reproduces the same sampling decisions. Change run_index to exercise a different deterministic run without changing the seed.

If you omit seed, Rightbrain generates one and returns it in chaos_config. Use that stored value for a replay. Approval resumes preserve the run’s existing policy, applied-fault count, latency budget, and event history.

Safety bounds

max_faults limits how many sampled faults can be applied. Tool latency is capped at 5 seconds per fault and 15 seconds across the run. Decisions that would exceed the fault or latency budget are retained as unapplied events with a reason.

These bounds prevent an unbounded policy, but they do not make a chaos run suitable for production traffic.

Inspect a chaos run

Agent run records expose:

• is_chaos_run — whether the run has an enabled policy.
• chaos_config — the normalized policy, including its replayable seed.
• chaos_events — every eligible sampling decision in execution order.

Each event records whether the fault was applied, its probability threshold and sampled value, the target and phase, and any mode-specific error, variant, retryability, or latency. Decision keys are returned as hashes.

Use chaos_enabled when listing agent runs to select chaos or non-chaos history.

Fallbacks and approvals

A provider_unavailable or other model fault exercises the same primary-model failure path as a provider incident. If the agent revision has an eligible fallback model, the run records the failover in its normal model-usage fields.

Tool faults continue through the standard tool-result pipeline. Approval-gated tools still follow their configured approval policy, and a resumed run continues with the original chaos budget and event record.

Related