---
title: "Agent Evaluations"
group: "Operate"
groupOrder: 4
order: 3
---
An agent evaluation in STAIR is a test that sends scripted requests to one agent and grades every answer with evaluators to give a pass rate.

## How Agent Evaluations work in STAIR

An agent evaluation, called a *test* in the console, ties together three things:

- **Target agent:** the agent under test. A test checks one agent, the same way users reach it in STAIR Chat or through the API.
- **Requests:** scripted dialogs. Each request has a **Dialog context**: one or more messages with the roles **User**, **Agent** or **System**, ending with the User message whose answer is graded.
- **Evaluations:** for each request, the evaluators that grade the answer, with their parameters and the expected target, for example the text fragments the answer must contain.

When you run a test, STAIR sends every request to the target agent, waits up to the **Question timeout** for each answer and applies the evaluators. A request passes when all its checks pass. The test passes when the share of passed requests is at least the **Minimum pass rate**. Each run is saved, so you can see whether the agent got better or worse after a change.

Evaluators come in several kinds:

| Kind | Examples | What they check |
|---|---|---|
| Text checks | Contains All, Contains Any, Not Contains, Forbidden Regex, Normalized Exact Match, Output Length | The text of the answer |
| Structure and numbers | JSON Schema, Numeric Match, Report Sections, Answer Format Contract | The format of the answer or a number in it |
| Tool calls | Tool Called, Not Tool Called, Tool Call Sequence, All Tool Arguments Match | Which tools the agent called, how often, in which order, and with which arguments |
| LLM judge | Factual Correctness, Answer Relevance, Answer Completeness, Answer Coherence, Instruction Adherence | The meaning of the answer, graded by a judge model |
| Performance | Latency Under Limit | The total response time |

The **Judge model** is optional. It becomes required as soon as a request uses an LLM-judge evaluator.

![The Agent Evaluations list filtered to the Demo test, with agent, judge model, pass rate and trend.](./agent-evaluations-test-list.png)

The list shows each test with its **Agent**, **Judge model**, **Pass rate / trend**, **Created by**, **Created at** and **Last run**. Use **Filter** to narrow the list by agent or judge model.

## How to create an agent evaluation

1. In the sidebar, open **Agent Evaluations** and click **Create**.
2. On the **General** tab, enter a **Test name** and a **Description**, and pick the **Target agent**. STAIR shows the agent's toolsets and tools, so you can plan tool-call checks.
3. Set **Question timeout** (seconds per request) and **Minimum pass rate** (%). Pick a **Judge model** if you plan to use LLM-judge evaluators.
4. Open the **Requests** tab and click **Add request**. Enter a **Request name**.
5. Under **Dialog context**, write the messages. Click **Add message** for each extra turn and use the role switch to set **User**, **Agent** or **System**. The last message must be a User message.
6. Under **Evaluations**, click **Add evaluator**, tick one or more evaluators in **Add evaluations** and click **Add**. Fill in the parameters and the expected value of each evaluator.
7. Repeat steps 4–6 for each request. Use **Copy request** to start a new request from an existing one.
8. On the **Access Control** tab, set the **Owner**, Read, Write and **Manage Access**, and **Execute Access** for the users and teams who may run the test.
9. Click **Create**.

To change a test, open its row menu (⋮) and click **Edit**. Then click **Update** to save or **Discard Changes** to cancel. **Copy** creates a new test with " (copy)" added to the name.

![The General tab of a test in edit mode with test name, target agent, judge model, question timeout and minimum pass rate.](./agent-evaluations-general-tab.png)

## How to choose evaluators for a request

1. Open **View all evaluators** at the top of the Agent Evaluations page. The **Evaluators** page lists every evaluator available on your installation.
2. Search by name and expand a row to see its **Editable parameters**, **Fixed parameters** and each parameter's **Default value**.
3. In a request, add the evaluators that match what you want to check. Combine a few simple checks with one LLM judge. For example, use **Contains Any** for an exact menu path, **Not Contains** for error words, and **Factual Correctness** with a reference answer.

![The Evaluators page filtered to "Contains", with the text and tool-call evaluators and their descriptions.](./agent-evaluations-evaluators.png)

Tool-call evaluators have extra settings such as **Call limits** (minimum and maximum calls) and **Argument checks**. LLM-judge evaluators let you pick a **Preset** or write a **Custom rubric**. Text evaluators have a **Case sensitive** switch.

![The Requests tab of the Demo test showing a dialog context and three evaluators with their parameters.](./agent-evaluations-requests-tab.png)

## How to run an agent evaluation and read the results

1. In the Agent Evaluations list, open the row menu (⋮) of the test and click **Run**. You can also open the test and use the ⋮ menu at the top right.
2. Open the test and go to the **Runs** tab. Every run shows its **Id**, **Started**, **Duration** and **Pass rate**.
3. Select a run. The right panel lists the requests with "All checks passed" or the number of failed checks.
4. Expand a request to see its **Dialog context**, the **Agent answer**, and for each evaluator the **Result** (pass or fail) and **Why**. The request also shows its latency.

To run a test, you need **Execute Access** to the test and to its target agent.

![A run of the Demo test with one request expanded: the agent answer and pass results with reasons for three evaluators.](./agent-evaluations-run-results.png)

## How to compare two runs of an agent evaluation

1. On the **Runs** tab, tick two runs and click **Compare**.
2. The **Compare runs** page shows **Baseline (A)** and **Current (B)**, the change in pass rate, and **Score over runs**.
3. The **Per-request diff** marks each request as **Fixed**, **Regressed**, **New**, **Removed**, **Changed** or unchanged. Select a request to see both answers and evaluator results side by side.
4. Click **View agent configuration changes** to see which agent fields changed between the two runs.

Use this after you change an agent's instructions, model or tools: run the test again and compare the new run with the previous one.

## Agent evaluation settings reference

### General tab

| Field | Required | Description |
|---|---|---|
| Test name | Yes | Name of the test. |
| Description | No | What the test checks and when to run it. |
| Target agent | Yes | The agent under test. |
| Judge model | When LLM-judge evaluators are used | The model that grades answers for LLM-judge evaluators. |
| Question timeout | Yes | Maximum time in seconds for the agent to answer each request. The default is 60. |
| Minimum pass rate | Yes | The test passes only when at least this percentage of requests pass. The default is 100. |

### Requests tab

| Field | Required | Description |
|---|---|---|
| Request name | Yes | Name of the request, shown in results. |
| Dialog context | Yes | The messages sent to the agent. It must contain at least one message and end with a User message. |
| Evaluations | Yes | At least one evaluator, each with its parameters, expected value and settings. |

### Access Control tab

| Field | Required | Description |
|---|---|---|
| Owner | Yes | The primary maintainer or team accountable for the test. |
| Read Access / Write Access | No | Users and teams that can view or edit the test. |
| Manage Access | Yes | Users and teams that can manage access and settings. |
| Execute Access | No | Users and teams that can run the test. They also need Execute Access to the target agent. |

## Troubleshooting Agent Evaluations

### "Judge model is required for the selected evaluations"

A request uses an LLM-judge evaluator such as Factual Correctness. Pick a **Judge model** on the **General** tab, or remove the LLM-judge evaluators.

### "Dialog context must end with a User message"

The last message of a request has the role Agent or System. Add a User message at the end, or switch the role of the last message to **User**.

### A request fails with a timeout

The agent needed longer than the **Question timeout**. Increase the timeout, or make the request simpler. Agents that call many tools need more time.

### "This tool is not available for the selected agent"

A tool-call evaluator refers to a tool that the target agent no longer has. Add the tool to the agent again, or pick another tool in the evaluator.

## FAQ

### Is the judge model required?

No. You need a judge model only when a request uses an LLM-judge evaluator. Text, tool-call and latency evaluators work without one.

### How is the pass rate calculated?

The pass rate is the share of requests in which all evaluators passed. The test passes when the pass rate is at least the **Minimum pass rate**.

### Can I test multi-turn conversations?

Yes. Add several messages to the **Dialog context** with the roles User, Agent and System. Only the answer to the last User message is graded.

### Can I check which tools the agent called?

Yes. Use tool-call evaluators such as **Tool Called**, **Not Tool Called**, **Tool Call Sequence** or **All Tool Arguments Match**.

### Can I run a test automatically?

You run a test from the console. To run agents or workflows on a timetable, see [Scheduled Agent Runs](/docs/scheduled-agent-runs/llm.md).

## Related

- [AI Agents](/docs/ai-agents/llm.md)
- [LLM Models](/docs/llm-models/llm.md)
- [Scheduled Agent Runs](/docs/scheduled-agent-runs/llm.md)
- [Security and Access Control](/docs/security-and-access-control/llm.md)
- [STAIR Concepts](/docs/stair-concepts/llm.md)