Production Controls

Evals

Check a task or agent against a set of cases before you ship a change.

Every change to a task or agent produces a new revision. Evals let you run that revision against a fixed set of cases before you make it active, so you catch regressions before they reach production rather than after.

Both tasks and agents support evals. The model is the same for each: you define an eval set (a reusable collection of cases), then trigger an eval run that executes the current configuration against every case in the set.

When to use evals

  • Pre-deploy regression check. Before activating a new task or agent revision, run your eval set against it to confirm existing behavior still holds.
  • Comparing prompts or models. Run the same eval set against different revisions to see which one you want to make active.
  • Guarding a known-good baseline. Keep a set of cases that represent behavior you never want to break, and run it whenever you touch the configuration.

Eval sets

An eval set is a named collection of cases scoped to a single task or agent. Its cases are drawn from existing runs, which you pass as task_run_ids when you create the set. You create it once and reuse it across revisions.

ActionTask endpointAgent endpoint
Create a setPOST …/task/{id}/eval/setPOST …/task-agent/{id}/eval/set
List / get setsGET …/task/{id}/eval/set[/{set_id}]GET …/task-agent/{id}/eval/set[/{set_id}]
Update a setPUT …/task/{id}/eval/set/{set_id}POST …/task-agent/{id}/eval/set/{set_id}
Delete a setDELETE …/task/{id}/eval/set/{set_id}DELETE …/task-agent/{id}/eval/set/{set_id}

All paths are under https://app.rightbrain.ai/api/v1/org/{org_id}/project/{project_id}.

Task eval sets are updated with PUT; agent eval sets are updated with POST. This is a deliberate difference between the two APIs, not a typo.

Eval runs

An eval run executes the task or agent against every case in a set and records the results.

ActionTask endpointAgent endpoint
Start a runPOST …/task/{id}/eval/runPOST …/task-agent/{id}/eval/run
List / get runsGET …/task/{id}/eval/run[/{run_id}]GET …/task-agent/{id}/eval/run[/{run_id}]

Run an eval set for a task

$curl -X POST \
>  https://app.rightbrain.ai/api/v1/org/$ORG_ID/project/$PROJECT_ID/task/$TASK_ID/eval/set \
>  -H "Authorization: Bearer $RB_API_KEY" \
>  -H "Content-Type: application/json" \
>  -d '{ "name": "Regression cases", "task_run_ids": ["<task-run-id-1>", "<task-run-id-2>"] }'

Poll GET …/task/{id}/eval/run/{run_id} to read the results once the run completes. The agent endpoints follow the same shape under …/task-agent/{id}/eval/.

How results are judged

An eval replays each case against the candidate revision and compares its output to the reference run, using an LLM-as-judge. The judge returns a binary verdict — pass or fail — for each case, plus its reasoning. Binary verdicts are deliberate: a clear pass/fail is more reliable and more actionable than a fuzzy numeric score.

  • Aggregate result. A run reports pass_count and fail_count across the set, so the release decision is a single number: did enough cases pass?
  • Cross-provider judge. The judge model records its name and provider. Judging with a different provider than the one that produced the output avoids a model grading its own work.
  • Metric deltas. Beyond correctness, each run compares the reference and candidate on token cost, credits, and timing (reference_avg, candidate_avg, delta_pct) — so you can see, for example, that a candidate revision passed every case but runs 50% slower before you decide to ship it.

As a release gate

Because an eval yields a pass rate, it drops straight into CI: block activation of a candidate revision unless it clears a minimum pass rate against your golden set. The candidate must differ from the reference for the comparison to be meaningful. This is the release gate that turns “I think this prompt is better” into “this revision passed 5 of 5 cases and costs 0.4% more.”

A typical pre-deploy flow

1

Assemble a set of cases

Create an eval set for the task or agent and add the cases you care about.

2

Make your change

Edit the task or agent. This creates a new revision but does not have to make it active yet. See Versioning & revisions.

3

Run the set against the new revision

Trigger an eval run and review the results.

4

Promote or discard

If the results hold, activate the revision. If not, keep the current active revision and iterate.

Versioning & revisions

How task and agent revisions are created and activated.

Observability & audit

Inspect the runs your evals produce.

Tasks

What a task is and how its revisions work.

Agents

What an agent is and how it uses the tools available to it.