Unified audio models capable of audio understanding, audio generation and, increasingly, audio editing are proliferating rapidly. However, each capability is evaluated in isolation on specialized benchmarks and a basic question remains unanswered: can these models understand the audio they generate/edit i.e. do they exhibit audio understanding-generation self-coherence (UGC_OWN)? We introduce TORUS, an audio-native unified benchmark for holistic, closed-loop evaluation of unified audio models comprising 1,053 questions (117 tests) spanning 21 tasks and 8 audio domain settings across all 3 audio types. We evaluate unified models and cascades of specialized models across objective and subjective metrics. Best unified models are able to answer only 52.4% questions about their generations, while gpt-audio can answer 53.6% questions. We hypothesize that a truly unified audio intelligence should be highly self-coherent and thus position understanding-generation coherence as an essential test for future unified audio systems. Further details are on our project page: https://torus-benchmark.github.io/

翻译：暂无翻译