Visually grounded speech systems learn from paired images and their spoken captions. Recently, there have been attempts to utilize the visually grounded models trained from images and their corresponding text captions, such as CLIP, to improve speech-based visually grounded models' performance. However, the majority of these models only utilize the pretrained image encoder. Cascaded SpeechCLIP attempted to generate localized word-level information and utilize both the pretrained image and text encoders. Despite using both, they noticed a substantial drop in retrieval performance. We proposed Segmental SpeechCLIP which used a hierarchical segmental speech encoder to generate sequences of word-like units. We used the pretrained CLIP text encoder on top of these word-like unit representations and showed significant improvements over the cascaded variant of SpeechCLIP. Segmental SpeechCLIP directly learns the word embeddings as input to the CLIP text encoder bypassing the vocabulary embeddings. Here, we explore mapping audio to CLIP vocabulary embeddings via regularization and quantization. As our objective is to distill semantic information into the speech encoders, we explore the usage of large unimodal pretrained language models as the text encoders. Our method enables us to bridge image and text encoders e.g. DINO and RoBERTa trained with uni-modal data. Finally, we extend our framework in audio-only settings where only pairs of semantically related audio are available. Experiments show that audio-only systems perform close to the audio-visual system.

翻译：视觉接地语音系统通过配对的图像及其口语描述进行学习。近期，已有研究尝试利用图像与对应文本描述训练的视觉接地模型（如CLIP）来提升基于语音的视觉接地模型的性能。然而，这些模型大多仅使用预训练的图像编码器。级联式SpeechCLIP尝试生成局部词级信息，并同时利用预训练的图像和文本编码器。尽管使用了两种编码器，但其检索性能仍出现显著下降。我们提出分段式SpeechCLIP，采用分层分段语音编码器生成类似单词单元的序列，并在这些单词级表示之上使用预训练的CLIP文本编码器，相较于级联变体SpeechCLIP取得了显著提升。分段式SpeechCLIP直接学习词嵌入作为CLIP文本编码器的输入，从而绕过了词汇嵌入层。本文通过正则化和量化方法探索将音频映射至CLIP词汇嵌入空间的方案。鉴于目标是将语义信息蒸馏至语音编码器，我们进一步探究使用大规模单模态预训练语言模型作为文本编码器的可行性。该方法能够桥接使用单模态数据训练的图像编码器（如DINO）与文本编码器（如RoBERTa）。最后，我们将框架扩展至仅包含语义相关音频对的纯音频场景。实验表明，纯音频系统的性能已接近视听系统。