Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have investigated complicated speech tokenization methods to discretize continuous speech signals so that language modeling techniques can be applied to speech data. However, existing approaches either model semantic tokens, potentially losing acoustic information, or model acoustic tokens, risking the loss of semantic information. Having multiple token types also complicates the architecture and requires additional pretraining. Here we show that discretizing mel-filterbank channels into discrete intensity bins produces a simple representation (dMel), that performs better than other existing speech tokenization methods. Using a transformer decoder-only architecture for speech-text modeling, we comprehensively evaluate different speech tokenization methods on speech recognition (ASR), speech synthesis (TTS). Our results demonstrate the effectiveness of dMel in achieving high performance on both tasks within a unified framework, paving the way for efficient and effective joint modeling of speech and text.

翻译：大型语言模型通过在海量文本数据上进行自监督预训练，彻底改变了自然语言处理领域。受此成功启发，研究者们探索了复杂的语音标记化方法，将连续语音信号离散化，从而使语言建模技术能够应用于语音数据。然而，现有方法要么建模语义标记，可能丢失声学信息；要么建模声学标记，存在语义信息丢失的风险。同时使用多种标记类型还会使架构复杂化，并需要额外的预训练。本文研究表明，将梅尔滤波器组通道离散化为离散强度区间，可以产生一种简单的表示（dMel），其性能优于其他现有的语音标记化方法。通过使用仅含解码器的Transformer架构进行语音-文本建模，我们在语音识别（ASR）和语音合成（TTS）任务上全面评估了不同的语音标记化方法。我们的结果证明了dMel在统一框架内实现两项任务高性能的有效性，为高效且有效的语音与文本联合建模铺平了道路。