Research
Personal context becomes an evaluation task in PersonaBench
Synthetic profiles and personal documents create a controlled setting for personalized retrieval and question answering.
A personalized assistant may need to connect information distributed across a person's documents and conversations. PersonaBench creates synthetic users and document collections to study that reasoning in a controlled setting.
Build the person before the documents
The data pipeline begins with a profile template that organizes demographic, preference, and social information. It then generates documents reflecting the synthetic person's background and activities.
This construction supplies both the material an assistant can retrieve and a reference for the personal information represented in that material. Researchers can ask questions about the user and examine how an answer relates to the underlying profile.
Follow information through retrieval and generation
PersonaBench evaluates retrieval-augmented generation pipelines on personalized questions. Retrieval selects relevant passages from the document collection, and a language model uses the retrieved context to produce an answer.
The paper also studies models given the full document collection. These experiments help examine the work involved in finding, extracting, and combining personal information across documents.
A testbed for contextual reasoning
The benchmark contributes a synthetic-data method and a shared evaluation setting for personalization research. The profile and document relationship makes the target information inspectable, while the questions exercise the assistant's ability to use that context.
Synthetic profiles form connected social communities. Generated document types include conversation histories, interactions with AI models, and purchase histories. Those records express attributes through activities and relationships, giving the evaluation a structured way to study how an assistant connects personal facts distributed across several kinds of source material.
Juntao Tan and collaborators developed PersonaBench at Salesforce AI Research. Weiran Yao is a co-author. The work appeared in Findings of ACL 2025. Read the publication for the profile-generation pipeline, evaluation design, and experimental analysis.