Evaluating tool-integrated agents with MCPEval

An agent evaluation can begin with the capabilities exposed by a tool server. MCPEval uses the Model Context Protocol as a common connection between those capabilities, the tasks built around them, and the trajectories produced by agents.

Generate tasks from an interactive interface

The framework combines task generation with a verification stage. Generated tasks are grounded in the operations available through MCP tools, allowing an evaluation to exercise concrete interactions with a domain environment.

That connection gives the evaluation an explicit surface to inspect. A tool has a defined interface, a task calls for particular work, and the agent produces a sequence of actions and observations while attempting it.

Use several views of the same run

MCPEval supports multiple performance metrics and analyzes agent trajectories in addition to completion. The paper evaluates models and prompting strategies across finance, accommodation, healthcare, sports, and national-park domains.

Its results reveal domain-specific performance patterns and differences between model configurations. A trajectory view helps connect those differences to the interaction: which tools were selected, how the model handled their responses, and how the work reached its final state.

Connect evaluation to integration work

The open framework makes task construction, agent execution, and scoring parts of a shared evaluation workflow. Researchers can inspect those components and extend the domains or metrics used in an experiment.

The evaluation includes strict and flexible tool-call matching, alongside model-judge rubrics for trajectories and final-response completion. These views expose different properties of a run: agreement with expected actions, the sequence of interaction, and the completed response. Keeping the scoring definitions visible helps explain how an agent receives its reported result.

Zhiwei Liu and collaborators developed MCPEval at Salesforce AI Research. Haolin Chen and Weiran Yao are co-authors. The work appeared in the EMNLP 2025 System Demonstrations track. Read the paper and explore the framework.