Evals

Changes to revisioned task or agent configuration produce a new revision. Evals let you run that revision against a fixed set of cases before you make it active, so you catch regressions before they reach production rather than after.

Both tasks and agents support evals. The model is the same for each: you define an eval set (a reusable collection of cases), then trigger an eval run that executes the selected candidate revision against every case in the set.

When to use evals

• Pre-deploy regression check. Before activating a new task or agent revision, run your eval set against it to confirm existing behavior still holds.
• Comparing prompts or models. Run the same eval set against different revisions to see which one you want to make active.
• Guarding a known-good baseline. Keep a set of cases that represent behavior you never want to break, and run it whenever you touch the configuration.

Eval sets

An eval set is a named collection of cases scoped to a single task or agent. Its cases are drawn from existing runs, which you pass as task_run_ids for a task set or task_agent_run_ids for an agent set when you create it. You create it once and reuse it across revisions.

All paths are under https://app.rightbrain.ai/api/v1/org/{org_id}/project/{project_id}.

Eval runs

An eval run executes the task or agent against every selected reference case and records the results.

Set candidate_revision_id to the draft revision you want to evaluate. If you omit it, the API selects an active revision; it does not automatically select your latest draft. The field works the same way for task and agent eval runs.

Choose reference cases in one of three ways:

• eval_set_id reuses a curated set.
• task_run_ids or task_agent_run_ids supplies an ad hoc list of up to 20 historical runs.
• last_n selects up to 20 recent eligible runs.

Use a fixed eval set for a release gate. Use explicit IDs for focused investigation, and last_n for development feedback when recent traffic is representative. Agent last_n selection excludes eval-generated and non-replayable runs. Task last_n selects recent successful runs, then rejects a case before dispatch if its replay metadata is invalid.

Run an eval set for a task

Save the created set’s id as EVAL_SET_ID and the draft revision’s id as CANDIDATE_REVISION_ID. Get the revision ID from the revisions array in the task’s update response; do not use the task’s top-level id. Read Versioning & revisions for the activation lifecycle.

curl -X POST \
  https://app.rightbrain.ai/api/v1/org/$ORG_ID/project/$PROJECT_ID/task/$TASK_ID/eval/set \
  -H "Authorization: Bearer $RB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "Regression cases", "task_run_ids": ["<task-run-id-1>", "<task-run-id-2>"] }'

Poll GET …/task/{id}/eval/run/{run_id} to read the results once the run completes. The agent endpoints follow the same shape under …/task-agent/{id}/eval/.

Read agent eval verdicts and metrics from the eval-run detail. Agent eval candidate runs are excluded from the normal agent run list and detail endpoints: fetching a result’s candidate_run_id through …/task-agent/{id}/run/{run_id} returns 404.

File-backed task cases

Task evals can replay reference runs that used direct image or PDF inputs. Rightbrain restores the processed input values from the reference run and loads the direct model-facing files from project storage for the candidate run.

Files produced by input processors are not uploaded again because their processed values are already present in the stored reference input. Rightbrain validates direct-file metadata before queueing the eval and fails the replay if a required stored file is missing or inaccessible.

How results are judged

An eval replays each case against the candidate revision and compares its output to the reference run, using an LLM-as-judge. The judge returns a binary verdict — pass or fail — for each case, plus its reasoning. Binary verdicts are deliberate: a clear pass/fail is more reliable and more actionable than a fuzzy numeric score.

• Aggregate result. A run reports pass_count and fail_count across the set, so the release decision is a single number: did enough cases pass?
• Cross-provider judge. The judge model records its name and provider. Judging with a different provider than the one that produced the output avoids a model grading its own work.
• Metric deltas. Beyond correctness, each run compares the reference and candidate on token counts, charged credits, and timing (reference_avg, candidate_avg, delta_pct) — so you can see, for example, that a candidate revision passed every case but runs 50% slower before you decide to ship it. The metrics.token_cost field contains total token counts, despite its name; it is not a currency amount.

Custom agent judge policy

An agent eval run can include judge_config with one objective and an ordered rubric of up to 20 criteria. Every criterion must pass for the case to pass. If you omit judge_config, the run uses the standard functional-equivalence rubric.

The resolved judge configuration is returned in agent eval-run detail. Its objective and rubric text are sent to the selected external judge-model provider and are visible to callers with access to the eval run.

As a release gate

First wait for the eval run to finish, then inspect its case verdicts, pass_count, fail_count, and error_count. A completed eval means evaluation finished; it does not mean the candidate passed.

Because an eval yields a pass rate, it drops straight into CI: block activation of a candidate revision unless it clears a minimum pass rate against your golden set. The candidate must differ from the reference for the comparison to be meaningful. This is the release gate that turns “I think this prompt is better” into “this revision passed 5 of 5 cases and costs 0.4% more.”

A typical pre-deploy flow

Related