In this work, we study the task of ``visually" translating scene text from a source language (e.g., English) to a target language (e.g., Chinese). Visual translation involves not just the recognition and translation of scene text but also the generation of the translated image that preserves visual features of the text, such as font, size, and background. There are several challenges associated with this task, such as interpolating font to unseen characters and preserving text size and the background. To address these, we introduce VTNet, a novel conditional diffusion-based method. To train the VTNet, we create a synthetic cross-lingual dataset of 600K samples of scene text images in six popular languages, including English, Hindi, Tamil, Chinese, Bengali, and German. We evaluate the performance of VTnet through extensive experiments and comparisons to related methods. Our model also surpasses the previous state-of-the-art results on the conventional scene-text editing benchmarks. Further, we present rigorous qualitative studies to understand the strengths and shortcomings of our model. Results show that our approach generalizes well to unseen words and fonts. We firmly believe our work can benefit real-world applications, such as text translation using a phone camera and translating educational materials. Code and data will be made publicly available.

翻译：本文研究了从源语言（如英语）到目标语言（如中文）进行“视觉化”场景文本翻译的任务。视觉翻译不仅涉及场景文本的识别与翻译，还包括生成保留文本视觉特征（如字体、大小和背景）的翻译图像。该任务存在若干挑战，例如对未见字符的字体插值、文本大小与背景的保留。为解决这些问题，我们提出了一种基于条件扩散的新方法——VTNet。为训练VTNet，我们构建了一个包含60万样本的合成跨语言场景文本图像数据集，涵盖英、印地、泰米尔、中、孟加拉、德等六种主要语言。通过大量实验和与相关方法的比较，我们评估了VTNet的性能。该模型在传统场景文本编辑基准测试中超越了先前的最优结果。此外，我们进行了严谨的定性研究以剖析模型的优势与不足。结果表明，我们的方法对未见词汇和字体具有良好的泛化能力。我们坚信该工作可惠及实际应用，例如手机相机文本翻译及教育材料翻译。代码与数据将公开发布。