Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelling techniques to audio data. However, traditional codecs often operate at high bitrates or within narrow domains such as speech and lack the semantic clues required for efficient language modelling. Addressing these challenges, we introduce SemantiCodec, a novel codec designed to compress audio into fewer than a hundred tokens per second across diverse audio types, including speech, general sound, and music, without compromising quality. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised pre-trained Audio Masked Autoencoder (AudioMAE), discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are used to reconstruct audio via a diffusion-model-based decoder. SemantiCodec is presented in three variants with token rates of 25, 50, and 100 per second, supporting a range of ultra-low bit rates between 0.31 kbps and 1.40 kbps. Experimental results demonstrate that SemantiCodec significantly outperforms the state-of-the-art Descript codec on reconstruction quality. Our results also suggest that SemantiCodec contains significantly richer semantic information than all evaluated state-of-the-art audio codecs, even at significantly lower bitrates. Our code and demos are available at https://haoheliu.github.io/SemantiCodec/.

翻译：大型语言模型（LLM）通过将音频转换为离散标记的音频编解码器，显著推进了音频处理领域，使得语言建模技术能够应用于音频数据。然而，传统编解码器通常在高比特率下运行，或局限于语音等狭窄领域，且缺乏高效语言建模所需的语义线索。为应对这些挑战，我们提出了SemantiCodec，这是一种新颖的编解码器，旨在将语音、通用声音和音乐等多种音频类型压缩至每秒少于一百个标记，同时不降低音质。SemantiCodec采用双编码器架构：语义编码器使用自监督预训练的音频掩码自编码器（AudioMAE），并通过对海量音频数据进行k均值聚类实现离散化；声学编码器则用于捕捉剩余细节。语义与声学编码器的输出通过基于扩散模型的解码器进行音频重建。SemantiCodec提供每秒25、50和100标记三种变体，支持0.31 kbps至1.40 kbps范围内的超低比特率。实验结果表明，SemantiCodec在重建质量上显著优于当前最先进的Descript编解码器。我们的研究还表明，即使在显著更低的比特率下，SemantiCodec所包含的语义信息也远比所有评估的先进音频编解码器更为丰富。代码与演示可在https://haoheliu.github.io/SemantiCodec/获取。