In this paper, we propose methods to build a powerful and efficient Image-to-Speech captioning (Im2Sp) model. To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp. We set the output of the proposed Im2Sp as discretized speech units, i.e., the quantized speech features of a self-supervised speech model. The speech units mainly contain linguistic information while suppressing other characteristics of speech. This allows us to incorporate the language modeling capability of the pre-trained vision-language model into the spoken language modeling of Im2Sp. With the vision-language pre-training strategy, we set new state-of-the-art Im2Sp performances on two widely used benchmark databases, COCO and Flickr8k. Then, we further improve the efficiency of the Im2Sp model. Similar to the speech unit case, we convert the original image into image units, which are derived through vector quantization of the raw image. With these image units, we can drastically reduce the required data storage for saving image data to just 0.8% when compared to the original image data in terms of bits. Demo page: https://ms-dot-k.github.io/Image-to-Speech-Captioning.

翻译：本文提出构建高效且强大的图像到语音字幕生成（Im2Sp）模型的方法。为此，我们首先将大规模预训练视觉-语言模型中与图像理解及语言建模相关的丰富知识迁移至Im2Sp模型。我们将所提出的Im2Sp模型的输出设置为离散化的语音单元，即自监督语音模型的量化语音特征。这些语音单元主要包含语言信息，同时抑制语音的其他声学特征。这使得我们能够将预训练视觉-语言模型的语言建模能力融合至Im2Sp的口语语言建模中。借助视觉-语言预训练策略，我们在两个广泛使用的基准数据库（COCO和Flickr8k）上取得了全新的Im2Sp最佳性能。随后，我们进一步提升了Im2Sp模型的效率。与语音单元类似，我们将原始图像转换为图像单元，通过向量量化原始图像得到。利用这些图像单元，我们可以将图像数据存储所需的数据量大幅降低至原始图像数据比特数的0.8%。演示页面：https://ms-dot-k.github.io/Image-to-Speech-Captioning