Most TextVQA approaches focus on the integration of objects, scene texts and question words by a simple transformer encoder. But this fails to capture the semantic relations between different modalities. The paper proposes a Scene Graph based co-Attention Network (SceneGATE) for TextVQA, which reveals the semantic relations among the objects, Optical Character Recognition (OCR) tokens and the question words. It is achieved by a TextVQA-based scene graph that discovers the underlying semantics of an image. We created a guided-attention module to capture the intra-modal interplay between the language and the vision as a guidance for inter-modal interactions. To make explicit teaching of the relations between the two modalities, we proposed and integrated two attention modules, namely a scene graph-based semantic relation-aware attention and a positional relation-aware attention. We conducted extensive experiments on two benchmark datasets, Text-VQA and ST-VQA. It is shown that our SceneGATE method outperformed existing ones because of the scene graph and its attention modules.

翻译：大多数TextVQA方法通过简单的Transformer编码器整合物体、场景文本和问题词汇，但这未能捕捉不同模态之间的语义关系。本文提出了一种基于场景图的协同注意力网络（SceneGATE）用于TextVQA，该网络揭示了物体、光学字符识别（OCR）标记和问题词汇之间的语义关系。通过构建基于TextVQA的场景图，该网络能够发现图像的潜在语义。我们设计了一个引导注意力模块，用于捕捉语言与视觉之间的模态内部交互，并将其作为模态间交互的引导。为显式教授两种模态之间的关系，我们提出并整合了两个注意力模块：基于场景图的语义关系感知注意力和位置关系感知注意力。我们在两个基准数据集Text-VQA和ST-VQA上进行了广泛实验。结果表明，由于场景图及其注意力模块的引入，我们的SceneGATE方法在性能上优于现有方法。