Multi-modal models have shown appealing performance in visual tasks recently, as instruction-guided training has evoked the ability to understand fine-grained visual content. However, current methods cannot be trivially applied to scene text recognition (STR) due to the gap between natural and text images. In this paper, we introduce a novel paradigm that formulates STR as an instruction learning problem, and propose instruction-guided scene text recognition (IGTR) to achieve effective cross-modal learning. IGTR first generates rich and diverse instruction triplets of <condition,question,answer>, serving as guidance for nuanced text image understanding. Then, we devise an architecture with dedicated cross-modal feature fusion module, and multi-task answer head to effectively fuse the required instruction and image features for answering questions. Built upon these designs, IGTR facilitates accurate text recognition by comprehending character attributes. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins. Furthermore, by adjusting the instructions, IGTR enables various recognition schemes. These include zero-shot prediction, where the model is trained based on instructions not explicitly targeting character recognition, and the recognition of rarely appearing and morphologically similar characters, which were previous challenges for existing models.

翻译：多模态模型近期在视觉任务中展现出令人瞩目的性能，因为指令引导训练激发了模型理解细粒度视觉内容的能力。然而，由于自然图像与文本图像之间的差距，现有方法无法直接应用于场景文本识别（STR）。本文提出一种将STR建模为指令学习问题的新范式，并设计指令引导场景文本识别（IGTR）方法以实现有效的跨模态学习。IGTR首先生成丰富多样的指令三元组<条件,问题,答案>，作为细粒度文本图像理解的引导信息。随后，我们构建了包含专用跨模态特征融合模块与多任务答案头的架构，有效融合所需指令特征与图像特征以回答问题。基于这些设计，IGTR通过理解字符属性实现精确文本识别。在英文与中文基准上的实验表明，IGTR以显著优势超越现有模型。此外，通过调整指令，IGTR可实现多种识别方案，包括零样本预测（模型基于未明确针对字符识别的指令进行训练）、罕见字符与形态相似字符的识别——这些此前是现有模型面临的挑战。