Research
Following a large-codebase task with LoCoBench-Agent
An interactive software-engineering benchmark connects repository exploration, tool use, error recovery, and architectural consistency.
Changing a large codebase involves a sequence of decisions: find the relevant files, understand their dependencies, make a change, and interpret the feedback. LoCoBench-Agent turns that sequence into an interactive evaluation environment for coding agents.
A repository becomes the working environment
The benchmark extends scenarios from LoCoBench into sessions where an agent can inspect files, search a project, and analyze code. Its tools give the model a way to acquire information as work progresses. The recorded interaction captures which evidence the agent sought and how it used the results.
Long sessions also require memory management. The framework compresses inactive files into structural summaries and summarizes older conversation history as context fills. This creates an explicit connection between the repository available to the agent and the information carried into its next decision.
Comprehension and effort have separate measures
Evaluation covers understanding, architectural consistency, and execution efficiency. The study examines how those measures change across context lengths and models. Its analysis identifies a trade-off between thorough exploration and the resources consumed by that exploration.
That structure makes the interaction itself useful evidence. Reading a file, recovering from an unsuccessful action, and revisiting an assumption each become parts of the engineering task that researchers can inspect.
Questions the benchmark makes concrete
The framework provides eight engineering tools and nine evaluation metrics across contexts ranging from 10,000 to one million tokens. Its compressed file summaries preserve function and class signatures, imports, and documentation. This makes context handling a defined part of the evaluated system, alongside the tools used to explore and change the repository.
Jielin Qiu and colleagues developed LoCoBench-Agent at Salesforce AI Research, with contributions from Haolin Chen and Weiran Yao. Read the paper.