Building verified tool-use training data

Tool-use training data should show how an agent completes a task. The request establishes intent. The tool contract defines the available operations. Execution supplies observations and changes the environment. Keeping these pieces together gives a training pipeline a clear object to generate, verify, and improve.

Begin data construction with a small executable task. Define the initial state, the information visible to the agent, and the result that counts as completion. This contract gives both the generator and the verifier a shared target, and it makes each example easier to inspect when the trained model later makes a similar mistake.

Write the outcome before the conversation

Take a referral-packet example in a simulated application. Referral 042 belongs to a particular patient, has a supporting consultation note available, and is waiting for preparation. The user asks the agent to attach the note and prepare the packet for review. The task contract names the referral, the correct document, and the resulting review-ready status.

The environment exposes tools to retrieve the referral, find its supporting documents, attach a document, and update the packet. A successful interaction reads the relevant records, associates the correct note, advances the packet, and reports the confirmed result. Those dependencies form a task blueprint from which several natural conversations can be generated.

The blueprint can also specify what the agent must learn along the way. One variant supplies the document identifier directly. Another requires a lookup. A third asks the agent to resolve an unclear document request with the user. Each variant adds a decision while preserving a verifiable outcome.

Verify structure, execution, and intent

Run structural checks first. The example should use valid roles, registered tool names, correctly typed arguments, and matching call identifiers. These checks catch malformed records early and produce precise feedback for regeneration.

Next, execute the proposed interaction against a controlled environment. Reset the initial state, replay the calls, and retain their observations. Check that the packet reached review-ready status and that its attached note belongs to the referral’s patient. Store the environment and tool versions with this evidence so another run can reproduce the check.

Semantic review then connects the result to the user’s request. In the worked example, a valid update to another referral would violate the task. A successful API response also needs a truthful final message that describes the packet actually prepared. Reviewing request, calls, observations, and final state together makes these relationships visible.

case_id: "referral-042"

document: "referral-order.pdf"

Vary the decisions within each task

Once the basic task replays, vary the conditions that change the correct behavior. Include several similar document titles, a note that requires a further lookup, a user who supplies a missing detail, or a tool response that calls for another action. Write the expected continuation for each condition and verify the resulting interaction.

Track this variation by behavior. A dataset ledger can record entity resolution, argument construction, dependency ordering, clarification, recovery, and final reporting. Sampling from that ledger gives the training mixture a deliberate shape. A large set of paraphrases then has a clear place alongside examples that teach a new decision.

Preserve the sequence of observations and actions when combining data from different environments. A common trajectory representation helps the training code handle varied sources, while source labels and explicit sampling weights keep the mixture inspectable.

Use model failures to choose the next examples

After training, examine failed trajectories and assign a concrete cause. If the agent repeatedly selects the first matching document, build examples that require disambiguation. If it reports completion before checking the tool response, add cases where the environment’s confirmation determines the answer.

Keep evaluation families separate from the generated training pool, retain a few replayable examples for each failure type, and compare the next checkpoint on both the target behavior and established tasks. This creates a practical loop from observed mistake to data intervention to measured behavior.