How to Evaluate AI Agents: Reliability, Cost, Latency, and Failure Modes

An AI agent can produce an impressive demo and still be a poor production system.

The reason is simple: a demo usually proves that the agent can complete a task. Production requires evidence that it can complete the task reliably, repeatedly, at an acceptable cost, within an acceptable time, and without unsafe side effects.

That changes how agents should be evaluated.

A single final answer is not enough. An agent acts across multiple steps, calls tools, changes state, encounters partial failures, and can reach the same result through different paths. The evaluation target is therefore not only the output. It is the whole trajectory.

OpenAI’s current agent evaluation guidance emphasizes traces, graders, datasets, and repeatable eval runs. Anthropic similarly describes agent evaluation as a multi-turn problem in which the transcript, tool calls, intermediate state, and final outcome all matter.

The practical objective is to answer four questions:

Did the agent complete the task?
Did it complete the task consistently?
How much time and money did it consume?
What kinds of failures does it produce when it does not succeed?

Start with task success, not model quality

Do not begin by asking whether the model is intelligent.

Begin by defining what success means in the environment where the agent operates.

For a coding agent, success may mean:

the requested behavior is implemented;
the relevant tests pass;
existing tests do not regress;
no unrelated files are modified;
the final explanation matches the actual diff.

For a support agent, success may mean:

the customer’s issue is resolved;
the correct policy was applied;
the correct backend state was changed;
escalation happened when required;
no unauthorized action was executed.

For a research agent, success may mean:

the answer addresses the question;
factual claims are supported by sources;
the sources are relevant and sufficiently authoritative;
citations resolve to the claims they support;
the agent does not fabricate evidence.

The evaluator should verify the real outcome whenever possible.

That is stronger than asking another model whether the response ‘looks good.’

Evaluate the final state and the trajectory separately

Two agents can arrive at the same correct result through very different processes.

One may:

select the right tool immediately;
make one valid call;
verify the result;
stop.

Another may:

call three irrelevant tools;
retry the same failed action;
expose unnecessary data;
eventually stumble into the correct final state.

If you grade only the final output, both runs may pass.

In production, they are not equivalent.

A useful evaluation therefore has at least two layers.

Outcome evaluation

Check whether the environment ended in the correct state.

Examples:

expected database values;
passing tests;
correct ticket status;
correct file contents;
correct generated artifact;
valid structured output.

Trajectory evaluation

Inspect how the agent reached that state.

Questions include:

Did it choose the right tools?
Were arguments correct?
Did it perform unnecessary actions?
Did it respect permissions?
Did it recover correctly from errors?
Did it stop once the task was complete?
Did it hand off or ask for approval at the right point?

OpenAI’s trace-grading guidance is designed for exactly this type of workflow-level inspection: traces capture model calls, tool calls, guardrails, and handoffs so they can be graded for regressions and behavioral failures.

Reliability requires repeated trials

Agent behavior is non-deterministic.

A task that succeeds once may fail on the next run even when the input is identical.

That means a single successful run tells you very little about reliability.

Run each important task multiple times.

Two useful metrics described by Anthropic are pass@k and pass^k.

pass@k asks whether at least one of k attempts succeeds.

This is useful when the product can tolerate retries or generate multiple candidates and choose one.

pass^k asks whether all k attempts succeed.

This is closer to the reliability requirement of customer-facing systems where the user expects the same task to work every time.

For production agents, consistency is usually more important than the best run the system can occasionally produce.

Measure cost per successful task

Raw token cost is not enough.

Suppose Agent A costs $0.08 per run and succeeds 60% of the time. Agent B costs $0.12 per run and succeeds 95% of the time.

Looking only at cost per run makes Agent A appear cheaper.

Looking at cost per successful task can reverse that conclusion.

The same principle applies to tool usage.

Track:

model input/output tokens;
number of model turns;
number of tool calls;
paid external API usage;
retry count;
sandbox or compute cost;
human-review cost where applicable.

Then normalize those numbers by successful completion.

An agent that is slightly more expensive per turn may be cheaper operationally if it reaches correct outcomes with fewer retries and less human intervention.

Latency is part of product quality

Agents often trade latency for quality because they plan, call tools, observe results, and iterate.

Measure more than total wall-clock time.

Useful latency metrics include:

time to first useful action;
model latency per turn;
tool latency;
number of sequential steps;
total task latency;
p50, p95, and p99 completion time.

Tail latency matters.

An agent that usually finishes in eight seconds but occasionally takes three minutes may create a worse product experience than one that consistently finishes in fifteen seconds.

Measure latency on the real environment, not only against mocked tools. Network behavior, rate limits, browser operations, cold starts, and external APIs can dominate total execution time.

Build a failure taxonomy

A useful eval suite does more than produce a score.

It tells you how the system fails.

Create explicit failure categories.

Planning failures

The agent chooses a poor strategy or decomposes the task incorrectly.

Tool-selection failures

It uses the wrong tool or misses a tool it should have used.

Argument failures

It chooses the correct tool but supplies invalid, incomplete, or dangerous arguments.

State-understanding failures

It misreads the environment, stale data, previous tool output, or current task state.

Recovery failures

A tool returns an error, but the agent retries blindly, loops, or abandons a recoverable task.

Completion failures

The work is already correct, but the agent continues acting and introduces regressions.

Permission failures

The agent attempts an action beyond the authority required for the task.

Communication failures

The underlying work is correct, but the user-facing answer misrepresents what happened or omits an important limitation.

Once failures are categorized, improvements become much more targeted.

A tool-selection problem may require clearer tool descriptions. A recovery problem may require explicit retry policy. A permission problem may require architecture changes rather than better prompting.

Use deterministic graders wherever possible

LLM-as-a-judge is useful, but it should not replace deterministic checks when the environment offers a source of truth.

For coding:

run tests;
run type checks;
run linters;
inspect changed files;
verify expected outputs.

For transactional systems:

inspect database state;
verify ledger entries;
check authorization state;
confirm idempotency.

For structured tasks:

validate JSON schemas;
check required fields;
compare exact values;
run business rules.

Use model-based graders for dimensions that are genuinely semantic or difficult to encode, such as explanation quality, research synthesis, or conversational quality.

The strongest evals often combine deterministic, model-based, and human grading.

Build the dataset from real failures

Do not create an eval set composed only of clean examples.

Start with:

real production failures;
edge cases found during development;
ambiguous requests;
malformed tool outputs;
permission boundaries;
adversarial or conflicting instructions;
tasks where previous versions regressed.

Every meaningful production failure should become a candidate regression test.

Over time, the eval set becomes an operational memory of what the system has learned not to break.

Separate capability from policy

An agent can be capable of completing a task and still be unacceptable because it violates a policy constraint.

Measure both.

For example, an agent may successfully issue a refund but fail because:

it skipped required approval;
the amount exceeded a configured limit;
it exposed sensitive account information;
it used a tool outside its permitted scope.

Task success and policy compliance should be distinct metrics.

Otherwise a system can appear to improve by becoming more aggressive.

Compare changes against a baseline

Every significant change should run against the same evaluation set:

model upgrade;
prompt change;
new tool;
new retrieval layer;
changed tool description;
routing change;
memory change;
permission change.

Compare:

task success;
consistency across repeated trials;
cost per success;
total latency;
tool-call count;
policy violations;
failure categories.

This avoids a common pattern in agent development: fixing one visible failure while silently degrading another part of the system.

A minimum production scorecard

Before shipping an agent, I would want at least this scorecard:

Dimension	Metric
Task success	% of tasks reaching verified correct state
Reliability	repeated-trial success / pass^k where relevant
Cost	average and p95 cost per successful task
Latency	p50 / p95 / p99 task completion time
Tool quality	correct tool and argument rate
Efficiency	turns and tool calls per successful task
Safety	permission/policy violation rate
Recovery	success rate after injected tool or environment failure
Regression	change vs. previous production baseline

No single number summarizes an agent well.

A benchmark score can tell you something about model capability. It does not tell you whether your specific agent, tools, prompts, permissions, data, and environment form a reliable product.

The design rule

Evaluate agents as systems.

Measure the final outcome, the trajectory, the operational cost, and the failure distribution.

Use deterministic ground truth whenever it exists. Run repeated trials. Keep real failures in the eval set. Compare every meaningful change against a stable baseline.

The goal is not to prove that an agent can succeed.

The goal is to know how often it succeeds, how it fails, what each success costs, and whether the system remains inside its intended boundaries while doing the work.