The development of audio foundation models has accelerated rapidly since the emergence of GPT-4o. However, the lack of comprehensive evaluation has become a critical bottleneck for further progress in the field, particularly in audio generation. Current audio evaluation faces three major challenges: (1) audio evaluation lacks a unified framework, with datasets and code scattered across various sources, hindering fair and efficient cross-model comparison;(2) audio codecs, as a key component of audio foundation models, lack a widely accepted and holistic evaluation methodology; (3) existing speech benchmarks are heavily reliant on English, making it challenging to objectively assess models' performance on Chinese. To address the first issue, we introduce UltraEval-Audio, a unified evaluation framework for audio foundation models, specifically designed for both audio understanding and generation tasks. UltraEval-Audio features a modular architecture, supporting 10 languages and 14 core task categories, while seamlessly integrating 24 mainstream models and 36 authoritative benchmarks. To enhance research efficiency, the framework provides a one-command evaluation feature, accompanied by real-time public leaderboards. For the second challenge, UltraEval-Audio adopts a novel comprehensive evaluation scheme for audio codecs, evaluating performance across three key dimensions: semantic accuracy, timbre fidelity, and acoustic quality. To address the third issue, we propose two new Chinese benchmarks, SpeechCMMLU and SpeechHSK, designed to assess Chinese knowledge proficiency and language fluency. We wish that UltraEval-Audio will provide both academia and industry with a transparent, efficient, and fair platform for comparison of audio models. Our code, benchmarks, and leaderboards are available at https://github.com/OpenBMB/UltraEval-Audio.

翻译：自GPT-4o问世以来，音频基础模型的发展迅速加速。然而，缺乏全面的评估已成为该领域进一步发展的关键瓶颈，尤其是在音频生成方面。当前的音频评估面临三大挑战：(1) 音频评估缺乏统一框架，数据集和代码分散在不同来源，阻碍了公平高效的跨模型比较；(2) 音频编解码器作为音频基础模型的关键组成部分，缺乏被广泛接受且全面的评估方法；(3) 现有的语音基准测试严重依赖英语，难以客观评估模型在中文上的性能。为解决第一个问题，我们引入了UltraEval-Audio，一个专为音频理解和生成任务设计的音频基础模型统一评估框架。UltraEval-Audio采用模块化架构，支持10种语言和14个核心任务类别，同时无缝集成了24个主流模型和36个权威基准测试。为提升研究效率，该框架提供一键评估功能，并配有实时公开排行榜。针对第二个挑战，UltraEval-Audio采用了一种新颖的音频编解码器综合评估方案，从三个关键维度评估性能：语义准确性、音色保真度和声学质量。为解决第三个问题，我们提出了两个新的中文基准测试——SpeechCMMLU和SpeechHSK，旨在评估中文知识掌握程度和语言流畅度。我们希望UltraEval-Audio能为学术界和工业界提供一个透明、高效、公平的音频模型比较平台。我们的代码、基准测试和排行榜可在https://github.com/OpenBMB/UltraEval-Audio获取。