Monitoring Evals
Score the real conversations your Agent or Workforce is having against Checks, on dashboards you configure per sample rate
The Monitor section continuously scores live tasks against Checks. Unlike the Test section, which runs simulated conversations, Monitor evaluates the real conversations your Agent or Workforce is having.
Monitor is organized into dashboards — you can create more than one (for example, one focused on tone, another on tool-use accuracy) and configure each independently.
Dashboards only score the latest published version of your Agent or Workforce, so a resource with no published version collects no data.
Creating a Monitor dashboard
-
Go to the Evaluate tab and select Monitor from the left sidebar.
-
Click + New dashboard and give it a name.
-
Attach one or more Checks from the Checks tab.
-
Set a Sample rate — the percentage of incoming tasks to evaluate.
-
Optionally set Filter tasks to control which task statuses trigger evaluations. Leave it blank to evaluate all tasks.
-
Save the dashboard.
Once configured, qualifying tasks are automatically scored at the sample rate you’ve set.
Viewing dashboard insights
Each Monitor dashboard shows:
| Metric | Description |
|---|---|
| Overall score | Aggregate score across all evaluated tasks in the selected date range |
| Total runs | Number of tasks evaluated |
| Checks | Which Checks are attached to the dashboard |
You also get:
- Overall score timeseries to spot regressions or improvements over time.
- Per-Check charts so you can see which criteria are slipping.
- Version markers that line up score changes with each publish.
- A list of evaluation runs with score, name, and a drill-in to the full conversation.
To adjust dashboard settings after initial setup, click the Settings button in the top right corner of the dashboard.