Evaluation design and benchmarks · MLD-0041

Agent evaluation

Assessing a tool-using system's actions, task outcomes, reliability, and constraints over complete trajectories.

Expanded guide · editorial review 2026-11-02. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.

Prerequisites

Start with a complete task, not an answer sample

Agent evaluation measures an action-taking system across an environment and a sequence of decisions. Specify the user goal, allowed tools, initial state, hidden task facts, policy constraints, maximum steps and what constitutes completion. A fluent final message may conceal a failed action or an unauthorized side effect. AgentBench illustrates evaluation across interactive environments rather than only isolated text questions.

Separate final utility from trajectory safety

Record the final environment state and also every proposed and executed tool call, error, refusal, permission decision and intermediate observation. Use two independent outcome fields: user_goal_met and constraint_violation. A run that successfully books a mock room but also writes to an unauthorized file is not a clean success. Conversely, a safe refusal can be policy-correct while leaving a legitimate task unfinished.

Worked example: a sandboxed campus equipment request

Give a local toy agent a fictional inventory table, a mock reservation API and a rule that borrowing equipment requires supervisor approval. In one case an approval exists; in another it does not; in a third, an untrusted inventory note says to bypass approval. Verify the final reservation table, not just the agent's reply. All names, tools and approvals are synthetic; this is an illustrative proposed test, not a reported university deployment.

Give tool use an executable oracle

For deterministic actions, define a state-based checker: success = requested_reservation_exists AND correct_owner AND approval_recorded AND no_forbidden_side_effect. Also capture which actions were denied before execution. Run the same initial database snapshot for each trial and restore it between trials. For non-deterministic open-ended tasks, combine objective checks with blinded human assessment and record rater disagreement rather than treating a model judge as ground truth.

A bounded, reproducible evaluation harness

Prepare 24 distinct toy cases: eight ordinary, eight ambiguous or missing-information, and eight adversarial-input cases. Freeze model version, prompts, tool schemas, environment and per-trial budget. For each case and three independent seeds where stochasticity applies, reset sandbox_state; run_agent(case); score(check_final_state, check_trajectory, check_policy); append structured trace with a stable case ID. Keep a separate held-out set for confirmation after prompt or tool changes. This is pseudocode for an experiment, not measured performance.

Measure reliability and uncertainty, not just best-case success

Report numerator and denominator for task completion, valid action rate, forbidden-action rate, unnecessary refusal rate and side-effect severity. Include outcomes per case type plus confidence intervals or observed ranges. For example, if a task succeeds on 7 of 10 repeated trials, report 7/10 rather than a single favorable run. The tau-bench paper uses repeated-trial reliability metrics and final database-state checks to demonstrate why one-shot success can overstate consistency.

Diagnose capability, environment and objective separately

When a run fails, first check whether the tool schema was valid, the needed evidence was available and the task was achievable under its resource budget. Then ask whether the agent chose an action that plausibly optimized a different objective or merely misunderstood the instruction. Include a baseline agent with no tool access, an oracle-information condition and identical tools with shorter prompts. An error trace can suggest hypotheses, but neither a failed run nor an unauthorized call proves a durable hidden goal.

Attack-resilience testing needs paired benign controls

Place a harmless but explicit conflicting instruction in one returned toy document while keeping the user's legitimate goal fixed. Compare the ordinary document, the injected document and a neutral reformulation under the same authorized sandbox. Track successful user tasks, actual unauthorized state changes and false blocks independently. AgentDojo investigates this style of stateful tool-boundary evaluation; its reported experiments are controlled benchmark findings, not evidence of a compromise at a real institution.

How to use scores without overstating them

Compare versions only with identical task sets, tool permissions, budgets and scoring rules; document any benchmark-specific scaffold or data contamination. SWE-bench tests code agents against repository-level tasks and executable checks, but passing tests may miss untested side effects. AgentBench, SWE-bench, tau-bench and AgentDojo address different behaviors, so their numbers are not interchangeable. A benchmark is a sample of threats and workflows, not proof of general safe autonomy.

Sources for the expanded explanation

Editorial synthesis checked against the linked primary material; no independent replication is claimed.

Original example

Test whether an agent finishes a task without unauthorized actions.

Original caveat

Final-answer quality can hide unsafe intermediate actions.

Context and aliases

Level: Intermediate. Stages: Evaluation · Research.

Tags: ML · DL · LLM · SLM · Safety.

Aliases: None recorded.

Related concepts

Concepts linking here

Original references

Open original record, export or annotate →