Text-based image captioning is an important but under-explored task, aiming to generate descriptions containing visual objects and scene text. Recent studies have made encouraging progress, but they are still suffering from a lack of overall understanding of scenes and generating inaccurate captions. One possible reason is that current studies mainly focus on constructing the plane-level geometric relationship of scene text without depth information. This leads to insufficient scene text relational reasoning so that models may describe scene text inaccurately. The other possible reason is that existing methods fail to generate fine-grained descriptions of some visual objects. In addition, they may ignore essential visual objects, leading to the scene text belonging to these ignored objects not being utilized. To address the above issues, we propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps. Concretely, to construct three-dimensional geometric relations, we introduce depth information and propose a depth-enhanced feature updating module to ameliorate OCR token features. To generate more precise and comprehensive captions, we introduce semantic features of detected visual object concepts as auxiliary information. Our DEVICE is capable of generalizing scenes more comprehensively and boosting the accuracy of described visual entities. Sufficient experiments demonstrate the effectiveness of our proposed DEVICE, which outperforms state-of-the-art models on the TextCaps test set. Our code will be publicly available.

翻译：基于文本的图像描述是一项重要但尚未充分探索的任务，旨在生成包含视觉对象和场景文本的描述。近期研究取得了令人鼓舞的进展，但仍面临场景整体理解不足及生成不准确描述的问题。一个可能的原因是当前研究主要构建场景文本的平面级几何关系而缺乏深度信息，导致场景文本关系推理不充分，模型可能对场景文本描述不准确。另一个可能原因是现有方法未能生成某些视觉对象的细粒度描述，且可能忽略关键视觉对象，导致这些被忽略对象对应的场景文本未被利用。针对上述问题，我们提出了一种深度与视觉概念感知Transformer（DEVICE）用于TextCaps任务。具体而言，为构建三维几何关系，我们引入深度信息并提出深度增强特征更新模块以改进OCR令牌特征。为生成更精确和全面的描述，我们引入检测到的视觉对象概念的语义特征作为辅助信息。我们的DEVICE能够更全面地泛化场景，提升描述视觉实体的准确性。充分实验证明了所提DEVICE的有效性，其在TextCaps测试集上的表现优于现有最优模型。我们的代码将公开。