How to evaluate an AI agent: metrics, eval harnesses and acceptance criteria

Most teams can get an agent demo working in an afternoon. The demo is not the hard part. The hard part is knowing whether the thing actually works — reliably, safely, and cheaply enough — across the messy reality of what your users will throw at it. That gap between “it worked in the demo” and “it works in production” is exactly where ai agent evaluation lives, and it is the single discipline that most often separates the agentic projects that ship from the ones that quietly get shelved.

The summary you can act on: before you write more agent logic, build a labelled dataset of real requests, define pass/fail acceptance criteria, and run that eval harness in CI. If you cannot measure task success, tool-call correctness, guardrail adherence, cost and latency on every change, you are not engineering an agent — you are gambling on one.

Why evaluation is the make-or-break for agentic AI

A traditional application is deterministic. Given the same input, you get the same output, so a handful of unit tests plus a QA pass gives you real confidence. An agent breaks that assumption in two ways. It is non-deterministic — the same request can produce different reasoning paths on different runs — and it is open-ended, choosing which tools to call, in what order, with what arguments. That combination means you cannot reason your way to confidence by reading the code. You have to observe behaviour across many inputs and measure it.

This is also why so many agent efforts stall. We wrote at length about the pattern in why most enterprise AI pilots fail, and evaluation sits near the centre of it. Teams build, demo, get excited, then hit production and discover the agent picks the wrong tool 12% of the time, hallucinates arguments on edge cases, and occasionally does something it was never meant to. Without an eval harness, they find this out from angry users rather than from a dashboard. With one, they find it before they ship — which is the whole point.

If you are choosing between an agent and a more rigid automation approach in the first place, that is a related but separate decision; our piece on AI agents versus RPA covers when the flexibility of an agent is worth the evaluation burden it creates. Assume here that you have made that call and committed to an agent. Now you have to prove it works.

What to measure

“Does it work” is not a metric. You need to decompose agent quality into things you can score independently, because an agent can be excellent on one axis and dangerous on another. Four categories cover most of what matters.

End-to-end task success

This is the headline number: given a real user request, did the agent achieve the intended outcome? Define success concretely per task type — the right record updated, the correct answer returned with a valid citation, the ticket routed to the right queue. Score it as a pass/fail or a graded rubric, not a vibe. Task success is the metric your business actually cares about, so it should be the one you optimise for, with everything else acting as a diagnostic when it drops.

For open-ended outputs where there is no single correct string, an LLM-as-judge — a separate model scoring the output against a rubric — is a practical way to grade at scale. Treat the judge with suspicion: calibrate it against a sample of human-labelled examples, keep its rubric explicit, and re-check it periodically. A miscalibrated judge gives you confident, wrong numbers, which is worse than no numbers.

Tool-call correctness

Agents act through tools, so most agent failures are really tool failures: the model called the wrong tool, called the right tool with wrong arguments, or called it at the wrong time. Measure this directly:

• Right tool selection — did it choose the correct tool for the request out of everything available?
• Argument correctness — were the parameters well-formed, grounded in the request, and within valid ranges? Fabricated arguments are a common and costly failure.
• Call sequencing — for multi-step tasks, did it call tools in a workable order, and did it stop when it should have rather than looping?

This is where a clean tool interface pays off. The clearer and more constrained your tool definitions, the easier they are both to call correctly and to score. If you are exposing tools over a standard protocol, our MCP implementation guide walks through the patterns that make tools legible to a model — and legible tools are, not coincidentally, evaluable tools.

Safety and guardrail adherence

Task success measures whether the agent did the right thing. Safety measures whether it avoided doing the wrong thing. These are not the same, and you must score them separately. Build eval cases specifically for the behaviours you want to prevent: acting outside its authority, leaking data it should not surface, taking an irreversible action without confirmation, or being talked out of its instructions by a hostile prompt.

For Australian organisations this is not optional polish. If your agent touches personal information, its behaviour has to be defensible under the Privacy Act and the APPs; regulated entities carry further obligations. Treat guardrail cases as a hard gate — a single failure on a safety-critical case should block the release regardless of how good the task-success numbers look. This is the practical, testable end of the governance conversation, and it is where an eval harness earns its keep with your risk and compliance stakeholders.

Cost and latency

An agent that is correct but takes 40 seconds and burns a small fortune in tokens per request is not shippable for most interactive use cases. Measure per-request cost (tokens in and out, across every model call in the trajectory) and latency (end-to-end, plus per-step so you can find the slow tool). Track the distribution, not just the average — the p95 tail is what users remember and what blows your budget. These numbers also make the business case legible; if you need to connect them to value, our AI ROI calculator and the CFO framework for measuring AI ROI give you the frame.

Building the eval harness

An eval harness is simply the machinery that runs your agent against a fixed set of inputs and scores the outputs on the metrics above. The engineering is not exotic. The discipline is.

Start with a labelled dataset of real requests. This is the asset that matters most, and there is no shortcut. Pull actual requests from your logs, support tickets, or a structured pilot, and for each one record the input, any relevant context, and the expected outcome or a rubric for judging it. A hundred well-chosen, genuinely representative cases beat a thousand synthetic ones. Deliberately over-sample the hard stuff: ambiguous phrasing, edge cases, the requests most likely to trigger a wrong tool call, and the adversarial inputs for your safety set. Synthetic data can pad coverage, but your dataset must be anchored in reality or your metrics will describe a world that does not exist.

Score each dimension explicitly. For every case, your harness should emit the task-success result, the tool-call assessment, the guardrail verdict, and the cost and latency figures. Log the full trajectory — every model call, tool call, and argument — because when a number drops, the aggregate score tells you that something broke and the trajectory tells you what.

Run it in CI. This is the step that turns evaluation from a one-off report into an engineering practice. Wire the harness into your continuous integration so it runs on every meaningful change — a tweaked prompt, a new tool, a reworked orchestration step. The build surfaces the scores; a regression on any gated metric fails the build. Once this is in place, your team stops arguing about whether a change helped and starts reading the numbers.

Setting acceptance criteria before you ship

Metrics without thresholds are just telemetry. Acceptance criteria are the explicit bars an agent must clear to go to production, and you should agree them before you start tuning, so the targets are honest rather than reverse-engineered to match whatever you happened to build. Sensible criteria look like:

• Task success at or above an agreed rate on the full dataset — set by what the workflow can tolerate, not by what looks impressive.
• Tool-call correctness above a high bar, because tool errors cascade into task failures and real-world side effects.
• Zero failures on safety-critical cases. No trade-offs here.
• Cost and latency within budgets tied to the use case and the ROI model.

Write these down and treat them as a release gate. If the agent misses, it does not ship — you fix it or you narrow the scope until it clears the bar. This is exactly the kind of measurable, time-boxed proof-of-value that a well-run generative AI pilot is built to deliver, and it is how you avoid the open-ended “keep tinkering” trap that sinks so many agent projects.

Regression testing when you upgrade the model

Here is the trap that catches teams who did everything else right: the model provider ships a new version, you upgrade for the better benchmarks and lower price, and your agent’s behaviour shifts underneath you. A newer model can be better on average and worse on your specific tool-calling patterns or your guardrails. Never swap the underlying model without re-running the full eval harness. Treat a model upgrade with the same rigour as a code change — run the dataset, compare against the previous version case by case, and hold the release to the same acceptance criteria. Your eval harness is precisely what makes an upgrade a measured decision instead of a leap of faith, and it is the difference between adopting a better model with confidence and discovering a regression through a production incident.

This is the connective tissue between a promising prototype and a system you can stand behind. If you are building agents and want the evaluation discipline built in from the start rather than bolted on after the first incident, our AI agents service is where that work sits. Have a look, and get in touch if you want a straight, no-hype conversation about what it takes to get an agent past the demo and into dependable production.