Where UserRL assigns credit in a conversation

A useful assistant learns which question to ask, when to consult a tool, and how to turn a conversation into a satisfactory outcome. UserRL provides a testbed for studying those decisions through reinforcement learning.

User interaction becomes a training environment

The framework pairs standardized gym environments with simulated users. An agent takes actions within a multi-turn exchange, observes responses, and receives feedback tied to the interaction. The environment and user simulator supply repeatable conditions for training experiments.

This organization lets researchers vary components of the interaction independently. They can change the model being trained, the simulator it talks to, or the reward used to score its behavior while keeping a common environment interface.

Credit can follow the sequence of decisions

UserRL separates turn-level reward shaping from trajectory-level scoring. The distinction makes credit assignment an explicit experimental choice. A question that helps a later decision can receive learning signal through reward-to-go scoring, which incorporates rewards from subsequent turns.

The study examines these scoring choices alongside supervised fine-tuning as an initialization step. Its experiments find that the initialization and trajectory reward design influence both task effectiveness and interaction efficiency.

Study the user side of the loop

The experiments also vary user simulators and evaluate transfer across them. This connects agent training to the behavior of the simulated person supplying goals, preferences, and replies. The released framework gives researchers a shared place to study that relationship.

For interactive-agent development, the testbed provides a practical vocabulary: an environment defines available actions, a user simulator shapes the conversation, and a reward specifies which outcomes the training process favors. Examining all three makes the learning setup concrete.

Cheng Qian and colleagues developed UserRL through Salesforce AI Research and the University of Illinois Urbana-Champaign, with Haolin Chen and Weiran Yao among the authors. Read the EMNLP 2026 paper.