Skip to main content
Evals is where you test and measure performance. It works on both Agents and Workforces, and lives in the Evaluate tab of the Agent or Workforce builder, next to Build and Run. Use it to create Test sets, define reusable scoring criteria, run evaluations, and monitor quality on live conversations.

​
Why use Evals?

Any time you put an Agent to work, you want to be sure it’s accomplishing your business outcomes. That might be increasing marketing qualified leads, or reducing the time it takes to respond to a lead. Evals gives you the confidence that your Agent is hitting the mark with its target audience — and that it still does after the next change. You write Tests that simulate the users and situations you care about, and Evals scores each one against success criteria you define. Agents and Workforces are non-deterministic — the same input can produce a different answer, and a change to a prompt, a Tool, or a model can shift behavior somewhere you weren’t looking — so holding the Tests and the criteria steady is what makes one version’s score comparable to the next.

Run Tests

Build Test sets holding Tests that simulate real user conversations, then attach criteria to score every one automatically.

Reuse criteria

Define a criterion once in the Checks tab and attach it to Tests, Monitor dashboards, or ad-hoc evaluations of completed tasks.

Monitor live tasks

Create Monitor dashboards that score live tasks against the same criteria, with sample-rate controls and per-criterion trend charts over time.

​
Overview of Evals

Evals has five core features, each with its own section in the left sidebar of the Evaluate tab:
  • Test — Create and manage Test sets. Each Test set holds Tests that simulate users; running a Test produces a conversation with the Agent or Workforce that gets scored by the criteria you attach.
  • Runs — Past evaluation run results. Browse average scores, tasks evaluated, progress status, cost (Credits and Actions), and creation date for every run.
  • Checks — Your library of scoring criteria; each one is called a Check. Create a Check once, then attach it to Tests, to Monitor dashboards, or to one-off evaluations of completed tasks.
  • Publish — Choose which Test sets must pass before your Agent or Workforce can be published. Set a minimum pass rate and optionally block publishing on failure.
  • Monitor — Track quality on real tasks. Create one or more Monitor dashboards, attach Checks, set a sample rate, and watch scores trend over time.

​
Cost and billing

Evaluations are billed in credits and actions. Each Test reports what it consumed in the results view.

​
Cost columns

Evaluation run results showing the Credits, Actions, and Run time columns for each Test

​
Cost breakdown

Clicking a Credits or Actions value opens a breakdown of where that Test’s cost went. It separates the user simulation, the Agent or Workforce’s own run, and the Checks that scored the conversation, so you can see which part drives the cost — usually its own execution. Cost breakdown modal splitting a Test's credits across Scenario Runner, Agent Execution, and Checks
Next: Learn how to create Tests — the simulated conversations your Agent gets evaluated against.

​
Frequently asked questions (FAQs)

Evals is available on every plan. The Evaluate tab sits next to Build and Run in the Agent or Workforce builder — if it isn’t there, contact support.
Evaluations consume both Actions and Vendor Credits. Each Check costs 1 Action, while Vendor Credits cover the LLM costs of the task run, the user simulator, and any LLM-based Check. Each Test shows its full breakdown in the results — see Cost columns above for details.