How to Evaluate AI Agents: From Sequential Tool Calls to End-to-End Task Completion

How to Evaluate AI Agents: From Sequential Tool Calls to End-to-End Task Completion Modern AI agent evaluation has shifted from scoring isolated function calls to assessing multi-step task completion across full execution environments. Measuring an agent requires tracking both process-level tool calls and end-to-end outcome state changes across dimensions of accuracy, verbosity, and operational cost. Isolated function-call evaluations like BFCL are insufficient because valid syntax fails if underlying workflow steps, checks, or updates are skipped. Comprehensive evaluation requires process scoring (step-level logs) for debugging and outcome scoring (end-to-end state verification) as a release gate. Benchmark runs follow a strict hierarchy of Benchmark, Trial, Task, Turn, and Step, prioritizing accuracy, verbosity, and cost metrics. Executable environmental verification provides the gold standard over static references and LLM judges, resisting emerging benchmark contamination vectors. NVIDIA Nemotron 3.5 Lightning demonstrates the approach, achieving 86% accuracy on PinchBench while finishing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy. Limitations of Static LLM Benchmarks Traditional model benchmarks were built for static inputs and decoupled protocols, which fail on agents executing multi-step workflows. While the Berkeley Function-Calling Leaderboard (BFCL) assesses function selection and argument accuracy across single- and multi-turn setups, evaluating individual calls remains insufficient. An agent can issue a syntactically correct call, such as issuing a refund, while failing the broader task by omitting critical prerequisites or updates. Process Scoring vs. End-to-End Outcome Scoring Full agent evaluation relies on a live execution environment that tracks state across sequential steps. Systems capture ordered logs known as traces, consisting of user inputs, agent actions, and intermediate environment states. Evaluations apply two distinct scoring layers: step-level process scoring grades whether individual tool calls were valid, relevant, and properly parameterized, pinpointing where execution broken down. End-to-end (E2E) outcome scoring ignores the intermediate path and gates releases solely on the final environment state, verifying whether actions like database updates or ticket resolutions actually occurred. Evaluation Hierarchy and Core Metrics Agent benchmark runs roll up through a fixed hierarchy: benchmark, trial (an independent pass over the task set), task (a scorable problem instance), turn (a conversational exchange), and step (an atomic tool invocation or command). Evaluation focuses on three core axes: accuracy, verbosity, and cost. Accuracy is tracked via task success rate, trial consistency across 3–5 passes, tool-call precision, and argument accuracy. Verbosity is measured by steps per success, while cost per success calculates the economic expenditure of tokens and compute per completed task. Benchmark Realism, Methodology, and Contamination Discrepancies across benchmarks arise from task complexity, statefulness, and verification methodology. Stateful environments expose context drift and corrupted state that static setups miss. Executable verification serves as the gold standard, whereas reference-based tests require maintenance, and LLM-as-a-judge scores remain provisional without human validation. Benchmark contamination has expanded to live retrieval by agents during runs and scraping from Hugging Face into pretraining sets, which private domain-specific benchmarks mitigate. Demonstration on SWE-bench and Nemotron 3.5 Lightning In a SWE-bench Verified run on an issue in pytest-dev__pytest-5262, an OpenHands agent reached an E2E score of 1 by passing real tests, while step-level scoring registered 3/4 due to a redundant file read that was subsequently recovered. Enterprise benchmark analysis similarly focuses on execution efficiency: on PinchBench, NVIDIA Nemotron 3.5 Lightning reached 86% accuracy while finishing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy, showing the advantage of optimizing speed and step efficiency alongside raw success rates.

AI Agent评估进阶：从单次工具调用到端到端任务完成 传统静态基准和孤立的函数调用测试无法衡量AI Agent在多步动态环境中的执行力，现代Agent评估已全面转向结合步骤级过程追踪与端到端环境状态验证的系统化指标体系。 单次函数调用正确无法保证任务成功，完整评估需依赖可追踪状态的执行环境，并划分为步骤级过程评分与端到端结果评分。 基准测试沿“基准-试验-任务-轮次-步骤”层级自底向上汇总，核心指标涵盖准确性、冗余度与单次成功成本三大维度。 可执行验证是衡量任务完成度的黄金标准，企业应建立基于真实工单、API与环境状态的私有域评估闭环。 评估重心演进：从单步调用到环境状态 传统LLM基准适用于静态任务，而BFCL等工具虽能评估单次调用的函数选择与参数精度，却无法判断是否遗漏前置检查或状态更新。现代Agent评估必须依托能够执行调用并追踪状态的完整环境。依托运行轨迹（Trace），评估分为两个层次：步骤级评分检查单步调用的有效性与相关性，用于调试和微调定位；端到端评分仅检验最终环境状态（如退款是否入账、工单是否流转），直接反映用户体验并作为上线门禁。 运行层级与核心度量指标三轴 基准测试按“基准→试验→任务→轮次→步骤”自底向上汇总，原子动作多为工具调用。评估指标收敛于三大维度：准确性（任务成功率、3至5次试验的一致性区间、工具调用精度、参数准确率）、冗余度（单次成功的步数）与成本（单次成功的消耗）。指标需组合观察，例如成功率脱离一致性仅是随机系统的点估计，而区分工具调用精度与参数准确率能有效隔离接口调用错误与槽位填充失败。 基准解读、实测轨迹与企业落地 在SWE-bench Verified的实际测试轨迹中，任务虽通过端到端测试获得E2E满分，但因检索代码时出现全文件导出的冗余操作，导致步骤级得分和工具调用精度降为3/4，参数准确率保持4/4。可执行验证相比LLM裁判更具客观性。面对模型能力评估，企业不应仅依赖公开抽象跑分，而应建立基于真实工单与环境状态的私有评测；例如Nemotron 3.5 Lightning在PinchBench上以86%的准确率将任务完成速度提升30%，展现出高完成效率在实际部署中的经济价值。

When you ship an AI agent , the key question is whether it can execute a chain of work across dozens of sequential tool calls against a live environment, and recover when a step fails. Scoring whether the model sounds right tells you almost nothing about whether the work finished . That gap is why agent evaluation has had to evolve from scoring a single function call to scoring an entire task, with tool calling as the connective tissue underneath. This post traces that arc and explains why nearly every serious agent benchmark now rests on tool use. Why isn’t standard LLM benchmarking enough? The original harnesses were built for static tasks. The first model-agnostic, open-source harness decoupled the model from the evaluation protocol. Agents broke this assumption. Operating across multi-step tasks, an agent calls tools, handles errors, and observes results over many steps, making a single output string insufficient. The Berkeley Function-Calling Leaderboard (BFCL) emerged to evaluate function selection and argument accuracy across single- and multi-turn scenarios. However, BFCL only evaluates individual calls —a valid issue_refund call still fails if underlying checks or updates were skipped. Call accuracy is necessary, but not sufficient. From scoring calls to scoring the environment Full agentic evaluation now requires a full execution environment : one that executes each tool call, tracks state across steps, and reads the world afterward to decide whether the work got done. Two scoring layers sit on top of it: Step-level ( process scoring ) asks was this call valid, relevant, and useful given the state at that point? End-to-end (E2E, or outcome scoring ) ignores the path and checks only the final state: did the refund post, did the ticket route correctly? Step-level tells you where the chain breaks, which is what you want when debugging or targeting fine-tuning effort; E2E collapses a failure on step one and a failure on step nine into the same “task failed.” E2E is what your users actually experience, which is why most production evals gate the release on it and keep step-level tracing underneath for debugging. Those two scores are two readings of one object: the trace . A trace is the ordered log of a single attempt: the user message, each step, and the environment state when the attempt stops. Process scoring grades the rows. E2E scoring grades the final state. What a benchmark run measures A tool-calling benchmark scores three things in order: deciding to use a tool, selecting the right one, and populating its arguments. A model that reaches for a tool when a direct answer would fail as surely as one that skips a tool it needed. Cost and latency ride on top, set by the call’s verbosity and runtime. Every run rolls up through a fixed hierarchy: Benchmark → Trial → Task → Turn → Step : A trial is one independent pass over the whole task set under a fixed configuration. A task is one independently scorable problem instance, identified by a task ID. A turn is one exchange boundary: a message in, the agent’s reply out, everything between belongs to that turn. A step is one atomic action inside a turn — a tool/command invocation, or a non-tool emission like a plan or the final message. Figure 1. Visualization of turn vs step in agent workflows A step is usually a tool call , and every score above it rolls up from those steps. The metrics worth tracking collapse onto three axes: accuracy , verbosity , cost (see Table 1, below). Metric Formula Axis Why it exists Task success rate successful_tasks / tasks Accuracy The release gate. Did the environment reach the goal state? Consistency range of success rate across 3–5 trials Accuracy A 90% / 74% split isn’t 84%. Report 82–88%, not a point estimate. Tool-call precision correct_calls / calls_issued Accuracy Hallucinated names and extra calls surface here, not in success rate. Argument accuracy correct_args / calls_with_right_tool Accuracy Separates “wrong API” from “right API, filled wrong.” Steps per success steps / successful_tasks Verbosity How long the trajectory runs when the task actually finishes. Cost per success spend / successful_tasks Cost The economic unit. Tokens and GPU-seconds only matter per successful task. Table 1. Core evaluation metrics, grouped by axis (accuracy, verbosity, cost) The pairings matter: success rate without consistency is a point estimate on a stochastic system (a model that hits 90% then 74% is a worse bet than one holding 84%); tool-call precision without argument accuracy hides slot-filling failures. Step count is often the axis that varies most across models on the same task — four steps versus fifteen — though on suites like Terminal-Bench 2.0 steps-per-turn varies too, so which axis moves most is benchmark-dependent. Parallel tool calling cuts step count and latency, but not call count: a one-step turn firing four tools still issued four calls. Roll up in order; don’t average steps and call it a benchmark score. How to read an evaluation Two benchmarks can both claim to test tool calling and produce numbers that aren’t comparable. Three dimensions explain most of the gap: Task complexity — single-turn with one tool, or multi-turn requiring planning, error recovery, and state management? A single-call benchmark won’t tell you whether a model collapses on step eight of fifteen. Statefulness — does the environment update on each action? Stateful benchmarks surface drift, context loss, and corrupted state that static ones miss. Methodology — executable verification (did the DB update, did tests pass) is the gold standard. Reference-based evaluation needs an annotated answer set someone must maintain. LLM-as-a-Judge fills the gap where no executable check exists, but treats its scores as provisional until validated against human ratings on a sample. Contamination now extends beyond training data leaks to live variants: web-searching agents retrieving answer keys during evaluation, and datasets on Hugging Face quickly re-scraped into pretraining corpora. Private domain evals solve this by being unable to scrape. Table 2, below, is a public trace from a real benchmark run using step-level and E2E scoring, where the suite rather than an artificial ticket provides the tools, user, and completion criteria. Suite: SWE-bench Verified (real GitHub issues, executable test verification) Task ID: pytest-dev__pytest-5262 (trial .2, turns 0–4) User / user-simulator opening: “Implement the necessary changes to the repository ( /testbed ) so that the requirements specified in the issue are satisfied” — the issue: _pytest.capture.EncodedFile reports mode rb+ (binary) from its underlying buffer, but its write() only accepts str , so external code that checks .mode (e.g. youtube-dl ) crashes when it writes bytes . Harness notes (tools exposed, max steps, parallel calling on/off): OpenHands agent harness; tools exposed: terminal , file_editor , task_tracker , finish ; parallel tool-calling off (one tool call per turn); repo state persists turn to turn (real filesystem + git, not a mock). Step Turn Call Environment observation Verdict Why (valid / useful / redundant / recovered / policy) 1 0 terminal(find /testbed -name "capture.py") Returns /testbed/src/_pytest/capture.py valid Locates the file named in the issue before editing anything 2 1 file_editor(view, capture.py) Dumps the full file (400+ lines) redundant File is large; grepping for the class first would have been more targeted 3 2 terminal(grep -n "EncodedFile" capture.py) Returns 422: return EncodedFile(...) / 425: class EncodedFile(object): recovered Corrects step 2’s inefficiency by narrowing straight to the relevant lines 4 3–4 file_editor(view, view_range=[420,450]/[450,470]) Shows EncodedFile.__init__ / __getattr__ , revealing it delegates .mode straight from the binary-mode buffer valid Pinpoints the exact root cause (unfiltered __getattr__ delegation) that step 5+ fixes Table 2. Extracted trace call on SWE-Bench verified evaluation E2E check (DB state / tests / ticket): PASSED E2E score (0 or 1): 1 Step-level score (passes / steps): 3/4 Tool-call precision: 3/4 Argument accuracy: 4/4 Looking at the results that were outputted, it is important to look at the last 5 bullets: E2E check, E2E score, step level score, tool-call precision, and argument accuracy . E2E check tells us that the related tests for the bug issues it sought to fix passed, meaning E2E score in this case is 1 (is_resolved: true). The next metric is the step level score that tells you how many of the steps the model took were actually needed. Looking at the score for the table above, step level is 3/4 due to one of the steps in this case being redundant – in particular step 2. In this case, it directly plays into the tool-call precision which also received a 3/4 due to the minor misstep. For this trace, our final metric argument accuracy saw that all arguments filled in correctly with no malformed arguments. Why benchmarks are converging on tool use The line between “calling a tool” and “completing a task” no longer holds: most benchmarks measuring general capability now also measure tool use because models aren’t run without tools in any viable deployment. A benchmark that withholds tool access scores a capability nobody ships. Not every benchmark makes the case. HumanEval runs generated Python against unit tests, providing executable verification, but no tool call and no environment to act on. SWE-bench is where the shift becomes clear: resolving a real GitHub issue means navigating a codebase, writing a patch, and passing the suite — file-read, search, and edit calls in sequence. The score measures the outcome, but the trajectory underneath consists entirely of tool calls. In many cases, then, the benchmarks you already run for general capability are already exercising tool use. That reframes the question that matters: Academic benchmarks measure a model’s capability ceiling in the abstract. Enterprise benchmarks answer the narrower, more useful question: can it do my job — your tasks, against your APIs, under your policies? The closer a benchmark sits to production, the more its score should weigh in your decision. Analyzing Nemotron 3.5 Lightning through this lens Read NVIDIA Nemotron 3.5 Lightning’s published suite as task completion and time-to-done, not isolated call accuracy. Banking scores completion across a multi-turn banking conversation — the refund trace at scale, not a single call. GDPval-AA v2 scores real agentic work from actual job outputs, judged pairwise by a panel of LLM judges with Elo anchored to a 1,000 human-expert baseline — the kind of human validation that keeps a judge score trustworthy. On PinchBench , Nemotron 3.5 Lightning hits 86% accuracy while finishing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy — a model that finishes efficiently beats one scoring higher on isolated accuracy while burning more steps and tokens . Figure 2. On PinchBench, Nemotron 3.5 Lightning reaches 86% accuracy while completing tasks up to 30% faster than Qwen3.6 35B at comparable accuracy Public scores are a great signal, but they shouldn’t be considered as a release gate. Adapting the model to your task and use-cases is as important as ever. Benchmarking your own workload Set a public floor. Run a published agentic suite; record success rate and its range across 3–5 trials. Build a domain eval from your real tickets, traces, and APIs. Gate on environment state — a database row, a merged PR, a closed ticket — not a judge’s opinion of the final message. Adapt the model and harness to that distribution. Re-measure success rate, consistency, steps per success, and cost per success. Keep step-level traces for debugging. Verify consequences in the environment, use judges for language, and use tool-call precision and argument accuracy to find where the chain breaks. To reproduce the published numbers, Nemotron ships reproducibility docs covering the configs behind the model card scores. Try it on build.nvidia.com , grab weights from Hugging Face , or follow the NIM guide . Tool calling in the realm of LLM benchmarking is the foundation that evaluations today rest on. Being able to build, read, and understand these evaluations is pertinent in making an informed decision for your use case. Going further Stay up to date on NVIDIA Nemotron by subscribing to NVIDIA news and following NVIDIA AI on LinkedIn , X , YouTube , and the Nemotron channel on Discord . Access open Nemotron Models on Hugging Face and a collection of NIM microservices and Developer Examples on build.nvidia.com .