Research
APIGen builds function-calling data through execution
Format checks, executable APIs, and semantic review give synthetic tool-use examples a structured verification process.
A function-calling example connects a natural-language request with an API action and its arguments. APIGen creates these examples through a pipeline that includes execution as part of data verification.
Generate across an executable API collection
The research assembles APIs spanning several categories and uses them to generate diverse tool-use requests. Each example pairs the request with the function calls intended to satisfy it.
The API collection defines the capabilities available to the generator. Its range of interfaces gives the pipeline different functions, argument structures, and task settings to exercise.
Check three properties of an example
APIGen applies a hierarchy of checks. Format checking examines the structure of an example. Actual function execution tests the proposed calls against the API implementation. Semantic verification reviews the relationship between the request and the resulting action.
These stages create distinct records of how an example was assessed. The pipeline can inspect a syntactic issue, an execution issue, or a mismatch with the intended request at the stage where it appears.
Connect data construction with model training
The work releases a collection of 60,000 function-calling examples and evaluates models trained on the data using the Berkeley Function-Calling Benchmark. The experiment connects a specific generation-and-verification process with the behavior learned by an action model.
For researchers constructing synthetic agent data, APIGen provides a clear unit of work: a request, the available API contract, the proposed call, and evidence from execution and semantic review. Keeping those elements together supports investigation of individual examples and of the collection used in a training run.
Zuxin Liu, Thai Hoang, and colleagues developed APIGen at Salesforce AI Research. Weiran Yao is a co-author. The paper appeared in the NeurIPS 2024 Datasets and Benchmarks track. Read the paper and visit the project.