There has been a growing interest in developing multimodal machine translation (MMT) systems that enhance neural machine translation (NMT) with visual knowledge. This problem setup involves using images as auxiliary information during training, and more recently, eliminating their use during inference. Towards this end, previous works face a challenge in training powerful MMT models from scratch due to the scarcity of annotated multilingual vision-language data, especially for low-resource languages. Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability. However, these are not directly applicable to MMT since they do not provide aligned multimodal multilingual features for generative tasks. To alleviate this issue, instead of designing complex modules for MMT, we propose CLIPTrans, which simply adapts the independently pre-trained multimodal M-CLIP and the multilingual mBART. In order to align their embedding spaces, mBART is conditioned on the M-CLIP features by a prefix sequence generated through a lightweight mapping network. We train this in a two-stage pipeline which warms up the model with image captioning before the actual translation task. Through experiments, we demonstrate the merits of this framework and consequently push forward the state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The code can be found at www.github.com/devaansh100/CLIPTrans.

翻译：近年来，开发能够通过视觉知识增强神经机器翻译（NMT）的多模态机器翻译（MMT）系统日益受到关注。该问题设置涉及在训练过程中将图像作为辅助信息使用，并在近期进一步推进至推理阶段消除图像依赖。为此，先前研究面临因标注多语言视觉-语言数据稀缺（尤其是低资源语言）而难以从头训练强大MMT模型的挑战。与此同时，面向NMT的多语言预训练模型与面向视觉-语言任务的多模态预训练模型（主要基于英语）大量涌现，展现出卓越的泛化能力。然而，这些模型无法直接应用于MMT，因其无法为生成任务提供对齐的多模态多语言特征。为解决该问题，我们提出CLIPTrans，该方法无需设计复杂MMT模块，仅通过适配独立预训练的多模态M-CLIP与多语言mBART实现目标。为对齐其嵌入空间，我们通过轻量级映射网络生成前缀序列，使mBART以M-CLIP特征为条件。采用两阶段流水线训练：先以图像描述任务预热模型，再执行实际翻译任务。实验表明，该框架具有显著优势，在标准基准测试中平均BLEU值提升+2.67，刷新了当前最优性能。代码获取地址：www.github.com/devaansh100/CLIPTrans。