Speech discrete representation has proven effective in various downstream applications due to its superior compression rate of the waveform, fast convergence during training, and compatibility with other modalities. Discrete units extracted from self-supervised learning (SSL) models have emerged as a prominent approach for obtaining speech discrete representation. However, while discrete units have shown effectiveness compared to spectral features, they still lag behind continuous SSL representations. In this work, we propose MMM, a multi-layer multi-residual multi-stream discrete units extraction method from SSL. Specifically, we introduce iterative residual vector quantization with K-means for different layers in an SSL model to extract multi-stream speech discrete representation. Through extensive experiments in speech recognition, speech resynthesis, and text-to-speech, we demonstrate the proposed MMM can surpass or on-par with neural codec's performance under various conditions.

翻译：语音离散表示因其优越的波形压缩率、训练时的快速收敛性以及与其他模态的兼容性，已在多种下游应用中证明其有效性。从自监督学习模型中提取的离散单元已成为获取语音离散表示的重要方法。然而，尽管离散单元相较于频谱特征已显示出有效性，其性能仍落后于连续的自监督学习表示。本研究提出MMM，一种从自监督学习模型中提取多层多残差多流离散单元的方法。具体而言，我们针对自监督学习模型的不同层级，引入基于K均值聚类的迭代残差向量量化，以提取多流语音离散表示。通过在语音识别、语音再合成与文本转语音任务上的大量实验，我们证明所提出的MMM方法在多种条件下能够超越或达到神经编解码器的性能水平。