Text image translation (TIT) aims to translate the source texts embedded in the image to target translations, which has a wide range of applications and thus has important research value. However, current studies on TIT are confronted with two main bottlenecks: 1) this task lacks a publicly available TIT dataset, 2) dominant models are constructed in a cascaded manner, which tends to suffer from the error propagation of optical character recognition (OCR). In this work, we first annotate a Chinese-English TIT dataset named OCRMT30K, providing convenience for subsequent studies. Then, we propose a TIT model with a multimodal codebook, which is able to associate the image with relevant texts, providing useful supplementary information for translation. Moreover, we present a multi-stage training framework involving text machine translation, image-text alignment, and TIT tasks, which fully exploits additional bilingual texts, OCR dataset and our OCRMT30K dataset to train our model. Extensive experiments and in-depth analyses strongly demonstrate the effectiveness of our proposed model and training framework.

翻译：文本图像翻译（TIT）旨在将图像中嵌入的源文本翻译为目标译文，具有广泛的应用场景和重要的研究价值。然而，当前TIT研究面临两个主要瓶颈：1）该任务缺乏公开可用的TIT数据集；2）主流模型采用级联式架构，容易受到光学字符识别（OCR）错误传播的影响。本文首先标注了一个中英文TIT数据集OCRMT30K，为后续研究提供便利。随后，我们提出一种基于多模态码本的TIT模型，能够将图像与相关文本关联，为翻译提供有效的补充信息。此外，我们设计了一个多阶段训练框架，涵盖文本机器翻译、图像-文本对齐和TIT任务，充分利用额外的双语文本、OCR数据集以及我们的OCRMT30K数据集来训练模型。大量实验和深入分析充分证明了所提模型和训练框架的有效性。