Research
Seven tool-use error patterns made inspectable with ToolScan
A diagnostic benchmark organizes tool-calling failures by the function, arguments, format, and action sequence involved.
A tool call contains several decisions: choose a function, supply its arguments, format the request, and decide whether another call is needed. ToolScan organizes failures at those boundaries into a diagnostic benchmark.
A vocabulary for tool-use errors
The benchmark characterizes seven error types. These cover insufficient API calls, incorrect argument values, incorrect argument names, incorrect argument types, repeated calls, incorrect function names, and invalid output format.
Each category points to a different part of the interaction. An argument-type error concerns the contract of a selected function. An insufficient-call error concerns the actions needed to complete the request. Naming these patterns makes a failure easier to locate within the sequence.
Evaluate queries with annotated action paths
ToolScan supplies 150 human-annotated queries spanning ten environment categories. The annotations include action paths that can satisfy a request. The evaluation framework examines model outputs using the shared error vocabulary and provides feedback tied to tool-calling criteria.
The study applies the benchmark to several language models and finds these patterns across their outputs. The resulting profile describes the kinds of tool-use mistakes observed in a model configuration.
Use a diagnosis to guide the next change
For an agent developer, the categories suggest concrete follow-up questions. A format error leads to inspection of the serialization contract. Repeated calls lead to inspection of the previous result and the agent's next action. An incorrect value leads back to the information used to fill an argument.
This is the contribution of a diagnostic evaluation: a shared language connects an observed failure to a part of the system that can be examined and changed. A subsequent run can then test whether that same pattern appears again.
Shirley Kokane and colleagues developed ToolScan at Salesforce AI Research. Weiran Yao is a co-author. Read the paper for the benchmark design and error analysis.