With the continuous development of Optical Character Recognition (OCR) and the expansion of application fields, text recognition in complex scenes has become a key challenge. Factors such as multiple fonts, mixed scenes and complex layouts seriously affect the recognition accuracy of traditional OCR models. Although OCR models based on deep learning have performed well in specific fields or similar datasets in recent years, the generalization ability and robustness of the model are still a big challenge when facing complex environments with multiple scenes. Furthermore, training an OCR model from scratch or fine-tuning all parameters is very demanding on computing resources and inference time, which limits the flexibility of its application. This study focuses on a fundamental aspect of mixed text recognition in response to the challenges mentioned above, which involves effectively fine-tuning the pre-trained basic OCR model to demonstrate exceptional performance across various downstream tasks. To this end, we propose a parameter-efficient mixed text recognition method based on pre-trained OCR Transformer, namely DLoRA-TrOCR. This method embeds DoRA into the image encoder and LoRA into the internal structure of the text decoder, enabling efficient parameter fine-tuning for downstream tasks. Experiments show that compared to similar parameter adjustment methods, our model DLoRA-TrOCR has the smallest number of parameters and performs better. It can achieve state-of-the-art performance on complex scene datasets involving simultaneous recognition of mixed handwritten, printed and street view texts.

翻译：随着光学字符识别（OCR）的持续发展及应用领域的扩展，复杂场景下的文本识别已成为关键挑战。多字体、混合场景及复杂布局等因素严重影响着传统OCR模型的识别精度。尽管近年来基于深度学习的OCR模型在特定领域或相似数据集上表现良好，但在面对多场景复杂环境时，模型的泛化能力与鲁棒性仍面临重大挑战。此外，从零开始训练OCR模型或微调全部参数对计算资源和推理时间要求极高，限制了其应用灵活性。针对上述挑战，本研究聚焦混合文本识别的基础问题，即如何通过有效微调预训练基础OCR模型，使其在下游任务中展现卓越性能。为此，我们提出了一种基于预训练OCR Transformer的参数高效混合文本识别方法，即DLoRA-TrOCR。该方法将DoRA嵌入图像编码器，并将LoRA嵌入文本解码器的内部结构，从而实现针对下游任务的高效参数微调。实验表明，与同类参数调整方法相比，我们的模型DLoRA-TrOCR参数数量最少且性能更优。在涉及手写体、印刷体及街景文本混合识别的复杂场景数据集上，该模型能够达到最先进的性能水平。