> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.rightbrain.ai/v-1/docs/production/evals/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.rightbrain.ai/_mcp/server.

# Evals

> Build eval sets and run them against Rightbrain tasks and agents to catch regressions before a new revision goes live.

Changes to revisioned task or agent configuration produce a new revision. Evals let you run that revision against a fixed set of cases before you make it active, so you catch regressions before they reach production rather than after.

Both **tasks** and **agents** support evals. The model is the same for each: you define an **eval set** (a reusable collection of cases), then trigger an **eval run** that executes the selected candidate revision against every case in the set.

## When to use evals

* **Pre-deploy regression check.** Before activating a new task or agent revision, run your eval set against it to confirm existing behavior still holds.
* **Comparing prompts or models.** Run the same eval set against different revisions to see which one you want to make active.
* **Guarding a known-good baseline.** Keep a set of cases that represent behavior you never want to break, and run it whenever you touch the configuration.

## Eval sets

An eval set is a named collection of cases scoped to a single task or agent. Its cases are drawn from existing runs, which you pass as `task_run_ids` for a task set or `task_agent_run_ids` for an agent set when you create it. You create it once and reuse it across revisions.

| Action          | Task endpoint                          | Agent endpoint                               |
| --------------- | -------------------------------------- | -------------------------------------------- |
| Create a set    | `POST …/task/{id}/eval/set`            | `POST …/task-agent/{id}/eval/set`            |
| List / get sets | `GET …/task/{id}/eval/set[/{set_id}]`  | `GET …/task-agent/{id}/eval/set[/{set_id}]`  |
| Update a set    | `PUT …/task/{id}/eval/set/{set_id}`    | `POST …/task-agent/{id}/eval/set/{set_id}`   |
| Delete a set    | `DELETE …/task/{id}/eval/set/{set_id}` | `DELETE …/task-agent/{id}/eval/set/{set_id}` |

All paths are under `https://app.rightbrain.ai/api/v1/org/{org_id}/project/{project_id}`.

> **Note**
>
> Task eval sets are updated with `PUT`; agent eval sets are updated with `POST`. This is a deliberate difference between the two APIs, not a typo.

## Eval runs

An eval run executes the task or agent against every selected reference case and records the results.

Set `candidate_revision_id` to the draft revision you want to evaluate. If you omit it, the API selects an active revision; it does not automatically select your latest draft. The field works the same way for task and agent eval runs.

Choose reference cases in one of three ways:

* `eval_set_id` reuses a curated set.
* `task_run_ids` or `task_agent_run_ids` supplies an ad hoc list of up to 20 historical runs.
* `last_n` selects up to 20 recent eligible runs.

Use a fixed eval set for a release gate. Use explicit IDs for focused investigation, and `last_n` for development feedback when recent traffic is representative. Agent `last_n` selection excludes eval-generated and non-replayable runs. Task `last_n` selects recent successful runs, then rejects a case before dispatch if its replay metadata is invalid.

> **Warning**
>
> An agent eval replays the candidate revision, including its external tools and Integrations. It can repeat writes, consume provider quota, or trigger other side effects. Run agent evals against isolated systems and data, or ensure every external action is safe to repeat or approval-gated.

| Action          | Task endpoint                         | Agent endpoint                              |
| --------------- | ------------------------------------- | ------------------------------------------- |
| Start a run     | `POST …/task/{id}/eval/run`           | `POST …/task-agent/{id}/eval/run`           |
| List / get runs | `GET …/task/{id}/eval/run[/{run_id}]` | `GET …/task-agent/{id}/eval/run[/{run_id}]` |

### Run an eval set for a task

Save the created set’s `id` as `EVAL_SET_ID` and the draft revision’s `id` as `CANDIDATE_REVISION_ID`. Get the revision ID from the `revisions` array in the task’s update response; do not use the task’s top-level `id`. Read [Versioning & revisions](/docs/production/revisions) for the activation lifecycle.

**`Create an eval set`**

```bash title="Create an eval set"
curl -X POST \
  https://app.rightbrain.ai/api/v1/org/$ORG_ID/project/$PROJECT_ID/task/$TASK_ID/eval/set \
  -H "Authorization: Bearer $RB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "Regression cases", "task_run_ids": ["<task-run-id-1>", "<task-run-id-2>"] }'
```

**`Trigger an eval run`**

```bash title="Trigger an eval run"
curl -X POST \
  https://app.rightbrain.ai/api/v1/org/$ORG_ID/project/$PROJECT_ID/task/$TASK_ID/eval/run \
  -H "Authorization: Bearer $RB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "eval_set_id": "'"$EVAL_SET_ID"'", "candidate_revision_id": "'"$CANDIDATE_REVISION_ID"'" }'
```

Poll `GET …/task/{id}/eval/run/{run_id}` to read the results once the run completes. The agent endpoints follow the same shape under `…/task-agent/{id}/eval/`.

Read agent eval verdicts and metrics from the eval-run detail. Agent eval candidate runs are excluded from the normal agent run list and detail endpoints: fetching a result’s `candidate_run_id` through `…/task-agent/{id}/run/{run_id}` returns `404`.

### File-backed task cases

Task evals can replay reference runs that used direct image or PDF inputs. Rightbrain restores the processed input values from the reference run and loads the direct model-facing files from project storage for the candidate run.

Files produced by input processors are not uploaded again because their processed values are already present in the stored reference input. Rightbrain validates direct-file metadata before queueing the eval and fails the replay if a required stored file is missing or inaccessible.

## How results are judged

An eval replays each case against the candidate revision and compares its output to the reference run, using an **LLM-as-judge**. The judge returns a binary `verdict` — `pass` or `fail` — for each case, plus its `reasoning`. Binary verdicts are deliberate: a clear pass/fail is more reliable and more actionable than a fuzzy numeric score.

* **Aggregate result.** A run reports `pass_count` and `fail_count` across the set, so the release decision is a single number: did enough cases pass?
* **Cross-provider judge.** The judge model records its `name` and `provider`. Judging with a different provider than the one that produced the output avoids a model grading its own work.
* **Metric deltas.** Beyond correctness, each run compares the reference and candidate on token counts, charged credits, and timing (`reference_avg`, `candidate_avg`, `delta_pct`) — so you can see, for example, that a candidate revision passed every case but runs 50% slower before you decide to ship it. The `metrics.token_cost` field contains total token counts, despite its name; it is not a currency amount.

### Custom agent judge policy

An agent eval run can include `judge_config` with one `objective` and an ordered `rubric` of up to 20 criteria. Every criterion must pass for the case to pass. If you omit `judge_config`, the run uses the standard functional-equivalence rubric.

The resolved judge configuration is returned in agent eval-run detail. Its objective and rubric text are sent to the selected external judge-model provider and are visible to callers with access to the eval run.

### As a release gate

First wait for the eval run to finish, then inspect its case verdicts, `pass_count`, `fail_count`, and `error_count`. A `completed` eval means evaluation finished; it does not mean the candidate passed.

Because an eval yields a pass rate, it drops straight into CI: block activation of a candidate revision unless it clears a minimum pass rate against your golden set. The candidate must differ from the reference for the comparison to be meaningful. This is the release gate that turns "I think this prompt is better" into "this revision passed 5 of 5 cases and costs 0.4% more."

## A typical pre-deploy flow

#### Assemble a set of cases

Create an eval set for the task or agent and add the cases you care about.

#### Make your change

Edit the task or agent. This creates a new revision but does not have to make it active yet. See [Versioning & revisions](/docs/production/revisions).

#### Run the set against the new revision

Trigger an eval run with the new revision’s ID in `candidate_revision_id` and review the results.

#### Promote or discard

If the results hold, activate the revision. If not, keep the current active revision and iterate.

## Related

#### [Versioning & revisions](/docs/production/revisions)

How task and agent revisions are created and activated.

#### [Observability & audit](/docs/production/observability)

Inspect the runs your evals produce.

#### [Chaos testing](/docs/production/chaos-testing)

Exercise bounded failure paths separately from quality regression cases.

#### [Tasks](/docs/concepts/tasks)

What a task is and how its revisions work.

#### [Agents](/docs/concepts/agents)

What an agent is and how it uses the tools available to it.

> Build reliable AI agents that run inside your existing tools and workflows. Rightbrain developer documentation.