Research
What UserBench measures about clarification
A controlled travel-planning environment tests whether an agent uncovers hidden preferences and selects suitable options.
A travel request can leave important details implicit: the pace of a trip, a preference for certain activities, or the trade-offs a traveler is willing to make. UserBench builds those hidden preferences into an interactive evaluation environment.
Make clarification part of the task
The environment gives an agent an initial request containing incomplete or indirect information about what the user wants. A simulated user can reveal additional preferences during the conversation. The agent must decide which clarifying questions will help it plan.
Structured option databases provide the tool-grounded part of the task. An agent searches available choices and selects options according to the preferences it has uncovered. The same interaction therefore connects language understanding, information gathering, and a concrete final selection.
Inspect the information gained in each turn
UserBench evaluates proactive preference elicitation alongside tool use and user alignment. Its analysis examines whether extra turns uncover useful information and how well the resulting plan reflects the hidden preferences.
The published experiments identify successful tool execution and preference discovery as separate capabilities. That separation gives researchers a way to trace a planning outcome to the questions and searches that preceded it.
A controlled setting for dialogue research
Built on Gymnasium, the testbed provides a standardized interaction interface and a stable tool backend. The setup makes it possible to compare agents under the same planning conditions and examine how their conversational strategies differ.
One useful lens is to treat a clarification question as an information-gathering action. Its value comes from the decision it enables: narrowing a search, resolving a trade-off, or choosing between options. UserBench makes that chain available for evaluation.
Cheng Qian and collaborators developed UserBench at Salesforce AI Research and the University of Illinois Urbana-Champaign. The team includes Haolin Chen and Weiran Yao. Read the EMNLP 2026 publication.