This paper presents an in-depth study of multimodal machine translation (MMT), examining the prevailing understanding that MMT systems exhibit decreased sensitivity to visual information when text inputs are complete. Instead, we attribute this phenomenon to insufficient cross-modal interaction, rather than image information redundancy. A novel approach is proposed to generate parallel Visual Question-Answering (VQA) style pairs from the source text, fostering more robust cross-modal interaction. Using Large Language Models (LLMs), we explicitly model the probing signal in MMT to convert it into VQA-style data to create the Multi30K-VQA dataset. An MMT-VQA multitask learning framework is introduced to incorporate explicit probing signals from the dataset into the MMT training process. Experimental results on two widely-used benchmarks demonstrate the effectiveness of this novel approach. Our code and data would be available at: \url{https://github.com/libeineu/MMT-VQA}.

翻译：本文对多模态机器翻译（MMT）进行了深入研究，探讨了“当文本输入完整时，MMT系统对视觉信息的敏感度会降低”这一普遍认知。相反，我们将此现象归因于跨模态交互不足，而非图像信息冗余。提出了一种新方法，从源文本生成平行的视觉问答（VQA）风格对，以促进更鲁棒的跨模态交互。利用大型语言模型（LLMs），我们显式地对MMT中的探测信号进行建模，将其转换为VQA风格数据，从而构建Multi30K-VQA数据集。引入了一个MMT-VQA多任务学习框架，将数据集中的显式探测信号融入MMT训练过程。在两个广泛使用的基准测试上的实验结果证明了该方法的有效性。我们的代码和数据将开源在：\url{https://github.com/libeineu/MMT-VQA}。