Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction. Joint speech-text modeling is a promising direction to achieve this. However, the effectiveness of recent speech tokens for joint modeling remains underexplored. To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through a attention-based aggregation mechanism and with speech reconstruction as the training objective. We conduct extensive experiments and show that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length. With TASTE, we perform straightforward joint spoken language modeling by using Low-Rank Adaptation on the pre-trained text LLM. Experimental results show that TASTE-based SLMs perform comparable to previous work on SALMON and StoryCloze; while significantly outperform other pre-trained SLMs on speech continuation across subjective and objective evaluations. To our knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to automatically learn a text-aligned speech tokenization and embedding suitable for spoken language modeling. Our demo, code, and model are available at https://mtkresearch.github.io/TASTE-SpokenLM.github.io.

翻译：近期研究致力于开发既能听又能说的口语语言模型（SLMs），以实现更自然的人机交互。联合语音-文本建模是实现这一目标的有效途径。然而，现有语音分词方法在联合建模中的有效性尚未得到充分探索。为此，我们提出了文本对齐语音分词与嵌入方法（TASTE），该方法通过在分词阶段将语音分词与对应文本转录对齐，直接解决模态差异问题。我们提出了一种基于注意力聚合机制并以语音重建为训练目标的方法来实现这一目标。我们进行了大量实验，结果表明TASTE在显著缩短分词序列长度的同时，能够有效保留关键的副语言信息。基于TASTE，我们通过对预训练文本大语言模型进行低秩适配，实现了简洁的联合口语语言建模。实验结果显示：基于TASTE的SLMs在SALMON和StoryCloze基准测试中达到了与先前工作相当的性能；同时在语音延续任务的主观与客观评估中显著优于其他预训练SLMs。据我们所知，TASTE是首个利用重建目标端到端自动学习适用于口语语言建模的文本对齐语音分词与嵌入方法。演示系统、代码及模型已发布于 https://mtkresearch.github.io/TASTE-SpokenLM.github.io。