In this paper, we propose a novel multi-modal framework for Scene Text Visual Question Answering (STVQA), which requires models to read scene text in images for question answering. Apart from text or visual objects, which could exist independently, scene text naturally links text and visual modalities together by conveying linguistic semantics while being a visual object in an image simultaneously. Different to conventional STVQA models which take the linguistic semantics and visual semantics in scene text as two separate features, in this paper, we propose a paradigm of "Locate Then Generate" (LTG), which explicitly unifies this two semantics with the spatial bounding box as a bridge connecting them. Specifically, at first, LTG locates the region in an image that may contain the answer words with an answer location module (ALM) consisting of a region proposal network and a language refinement network, both of which can transform to each other with one-to-one mapping via the scene text bounding box. Next, given the answer words selected by ALM, LTG generates a readable answer sequence with an answer generation module (AGM) based on a pre-trained language model. As a benefit of the explicit alignment of the visual and linguistic semantics, even without any scene text based pre-training tasks, LTG can boost the absolute accuracy by +6.06% and +6.92% on the TextVQA dataset and the ST-VQA dataset respectively, compared with a non-pre-training baseline. We further demonstrate that LTG effectively unifies visual and text modalities through the spatial bounding box connection, which is underappreciated in previous methods.

翻译：本文提出了一种新颖的多模态框架用于场景文本视觉问答（STVQA），该任务要求模型能够读取图像中的场景文本来进行问答。与可以独立存在的纯文本或视觉对象不同，场景文本同时传递语言语义并作为图像中的视觉对象，自然地连接了文本与视觉两种模态。与传统的STVQA模型将场景文本的语言语义和视觉语义作为两个独立特征不同，本文提出了一种"定位然后生成"（LTG）范式，通过空间边界框作为桥梁显式统一这两种语义。具体而言，首先，LTG通过答案定位模块（ALM）定位图像中可能包含答案单词的区域，该模块由区域提议网络和语言精炼网络组成，两者可通过场景文本边界框建立一一映射实现相互转换。其次，基于ALM选择的答案单词，LTG通过基于预训练语言模型的答案生成模块（AGM）生成可读的答案序列。得益于视觉与语言语义的显式对齐，即使没有基于场景文本的预训练任务，LTG在TextVQA数据集和ST-VQA数据集上相较于无预训练基线分别获得了+6.06%和+6.92%的绝对准确率提升。我们进一步证明LTG通过空间边界框连接有效统一了视觉与文本模态，而这一连接在之前的方法中未得到充分重视。