Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available: https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .

翻译：语音语言模型仅处理和生成声学数据，无需文本监督。在本文中，我们提出TWIST方法，该方法通过从预训练的文本语言模型进行热启动来训练语音语言模型。我们使用自动评估和人工评估表明，TWIST在各方面均优于冷启动语音语言模型。我们通过实验分析了不同模型设计选择（如语音分词器、预训练文本模型及数据集规模）的影响。研究发现，模型规模和数据集规模在构建性能更优的语音语言模型中均起着重要作用。基于上述观察，我们提出了据我们所知在参数量和训练数据规模上均为最大的语音语言模型。此外，我们引入了两个口头版本的StoryCloze文本基准测试，以进一步改进模型评估并推动该领域的未来研究。我们将语音样本、代码和模型公开提供：https://pages.cs.huji.ac.il/adiyoss-lab/twist/。