AudioCapBench examines what multimodal models hear

An audio description can carry many kinds of information: the identity of a sound, the instruments in a piece of music, or the emotional qualities of a speaker’s voice. Evaluating an assistant that hears these inputs requires a way to examine the details it chooses to describe.

AudioCapBench brings those details into a shared evaluation. The benchmark covers environmental sound, music, and speech, and compares native audio models using reference captions and a structured judging protocol.

Build a view across three audio domains

The evaluation set contains 1,000 clips: 400 environmental sounds, 300 music samples, and 300 speech samples. The sources include Clotho, AudioCaps, MusicCaps, and an emotional speech-caption dataset. Fixed-seed sampling makes the selected evaluation set reproducible.

Each domain asks the captioner to attend to different attributes. Environmental clips emphasize events and their acoustic setting. Music captions cover instrumentation, tempo, genre, and mood. Speech captions include speaker characteristics, emotion, speaking style, and spoken content. Keeping these domains visible helps an evaluator connect a score to the capability being exercised.

Separate the qualities of a caption

The primary judge assigns scores for accuracy, completeness, and hallucination control. Accuracy examines the correctness of described details. Completeness examines coverage of important reference content. The hallucination score rewards descriptions grounded in the supplied references. Category-specific prompts guide these judgments for each kind of audio.

The benchmark also calculates METEOR, BLEU-4, and ROUGE-L. These reference-based metrics add a reproducible textual view of the output. Together, the score dimensions let researchers inspect whether a model’s caption is accurate, sufficiently detailed, and grounded.

Use the domain breakdown to choose experiments

The paper evaluates 13 models from OpenAI and Google. In this comparison, Gemini 3 Pro records the highest overall judge score at 6.00 out of 10. The results also expose different model profiles across sound, music, and speech, with music a demanding category across the evaluated systems.

For teams developing multimodal applications, this breakdown makes the next experiment more specific. An application that depends on sound-event recognition can inspect those examples and scores directly. A speech-focused system can examine descriptions of vocal delivery alongside transcript content. Reusing the published data and scripts makes model comparisons easier to repeat as the application and its requirements change.

Research by Jielin Qiu, Haolin Chen, and collaborators at Salesforce AI Research.

Source: AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech.