Research
Building verifiable reinforcement-learning tasks with Webscale-RL
Domain-aware generation and verification turn pretraining documents into question-and-answer tasks for reinforcement learning.
Reinforcement learning needs tasks with feedback that a training system can use. Webscale-RL develops a data pipeline that converts large collections of documents into verifiable questions and answers across multiple subject areas.
Generate from a document and a perspective
The pipeline filters source material and assigns domain information. A generation stage conditions on the document, its domain, and a selected persona. That persona supplies a perspective from which to ask useful questions about the same material.
Domain-specific demonstrations guide the construction of question-and-answer pairs. The resulting tasks capture information from the source in a form that a model can attempt and a reward process can assess.
Build verification into the data engine
Quality checks are part of the construction process. They connect generated tasks to answerable material and prepare the examples for reinforcement learning. The released dataset contains 1.2 million examples across more than nine domains.
The research compares reinforcement learning on this data with continual pretraining and data-refinement baselines. It reports stronger benchmark performance and studies the amount of training data needed to reach comparable results.
A reusable path from information to tasks
The central engineering contribution is the connection between document processing and reward-bearing training examples. Document selection controls the material available to learn from; question generation controls the skill being exercised; verification supplies the feedback used in training.
Thinking about these stages separately also makes a data pipeline easier to investigate. A developer can trace an example back to its source, inspect the question produced from it, and examine the answer used during optimization. Those records give a concrete basis for studying how data construction affects a learning run.
Zhepeng Cen and colleagues developed Webscale-RL at Salesforce AI Research and Carnegie Mellon University, with Haolin Chen and Weiran Yao among the authors. The work appeared at ICLR 2026. Read the paper.