Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of these systems as well. We focus on text--audio and introduce Whisbert, which is inspired by the text--image approach of FLAVA (Singh et al., 2022). In accordance with Babylm guidelines (Warstadt et al., 2023), we pretrain Whisbert on a dataset comprising only 100 million words plus their corresponding speech from the word-aligned version of the People's Speech dataset (Galvez et al., 2021). To assess the impact of multimodality, we compare versions of the model that are trained on text only and on both audio and text simultaneously. We find that while Whisbert is able to perform well on multimodal masked modeling and surpasses the Babylm baselines in most benchmark tasks, it struggles to optimize its complex objective and outperform its text-only Whisbert baseline.

翻译：训练于多种输入模态可增强语言模型的能力。本文探究此类训练机制是否也能提升这些系统的质量与效率。我们聚焦于文本-语音任务，受FLAVA（Singh等，2022）文本-图像方法的启发，提出Whisbert模型。依据Babylm指南（Warstadt等，2023），我们在仅包含1亿词及其对应语音的数据集（基于词对齐版People's Speech数据集，Galvez等，2021）上预训练Whisbert。为评估多模态的影响，我们对比了仅文本训练与文本-语音联合训练的模型版本。实验表明，虽然Whisbert在多模态掩码建模中表现良好，并在多数基准测试中超越Babylm基线，但其在优化复杂目标函数及性能上仍难以超越纯文本训练的Whisbert基线模型。