Research
Retroformer trains an agent’s retrospective feedback
A learned reflection model uses environment rewards to turn previous attempts into guidance for the next run.
After an unsuccessful attempt, an agent can use feedback to revise its plan. Retroformer makes the quality of that feedback a learning problem by training a retrospective model with rewards from the environment.
Separate execution and reflection
The architecture includes an acting language model and a retrospective model. The actor attempts the task. The retrospective component examines the attempt, summarizes the cause of failure, and proposes guidance for a subsequent trial.
This division creates an explicit interface between what happened and what should change. A reflection becomes an input to the next attempt, where its usefulness can be assessed through the resulting task outcome.
Train the retrospective component
Retroformer uses policy-gradient optimization to update the model producing retrospective guidance. The actor participates in the environment that supplies feedback to this learning process. Training focuses on the component that writes the reflection.
The method can use reward information from different tasks and environments. A reflection's training signal is connected to the later execution it influences, giving the system a way to improve its revision behavior through experience.
Evaluate the loop across tasks
The experiments cover HotPotQA for search-based question answering, ALFWorld for text-based embodied tasks, and WebShop for browser-based shopping. They report improvements from the learned retrospective component. The released implementation makes the actor, feedback, and optimization loop available for further experimentation.
For agent researchers, the architecture provides a specific way to study reflection. They can inspect a failed trajectory, the guidance generated from it, and the actions taken during the next attempt. That sequence turns a general instruction to reflect into a component with inputs, outputs, and an outcome-linked learning objective.
Weiran Yao and collaborators developed Retroformer at Salesforce AI Research. The first report appeared in 2023, followed by publication at ICLR 2024. Read the paper and explore the implementation.