Layout-aware pre-trained models has achieved significant progress on document image question answering. They introduce extra learnable modules into existing language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. However, extra modules necessitate pre-training on extensive document images. This prevents these methods from directly utilizing off-the-shelf instruction-tuning language foundation models, which have recently shown promising potential in zero-shot learning. Instead, in this paper, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks. Based on this observation, we propose the LAyout and Task aware Instruction Prompt (LATIN-Prompt), which consists of layout-aware document content and task-aware instruction. Specifically, the former uses appropriate spaces and line breaks to recover the layout information among text segments obtained by OCR tools, and the latter ensures that generated answers adhere to formatting requirements. Moreover, we propose the LAyout and Task aware Instruction Tuning (LATIN-Tuning) to improve the performance of small instruction-tuning models like Alpaca. Experimental results show that LATIN-Prompt enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering, and LATIN-Tuning enhances the zero-shot performance of Alpaca significantly. For example, LATIN-Prompt improves the performance of Claude and ChatGPT on DocVQA by 263% and 20% respectively. LATIN-Tuning improves the performance of Alpaca on DocVQA by 87.7%. Quantitative and qualitative analyses demonstrate the effectiveness of LATIN-Prompt and LATIN-Tuning. We provide the code in supplementary and will release it to facilitate future research.

翻译：布局感知预训练模型在文档图像问答任务上取得了显著进展。这些模型通过在现有语言模型中引入额外可学习模块，利用OCR工具获取的文本边界框坐标来捕捉文档图像中的布局信息。然而，额外模块需要在大量文档图像上进行预训练。这阻碍了这些方法直接利用现成的指令调优语言基础模型，而后者近期在零样本学习中展现出巨大潜力。相反，本文发现Claude和ChatGPT等指令调优语言模型可以通过空格和换行符来理解布局。基于此观察，我们提出布局与任务感知指令提示（LATIN-Prompt），它由布局感知文档内容和任务感知指令组成。具体而言，前者使用适当的空格和换行符还原OCR工具获取的文本片段间的布局信息，后者确保生成的答案符合格式要求。此外，我们提出布局与任务感知指令调优（LATIN-Tuning）以提升Alpaca等小型指令调优模型的性能。实验结果表明，LATIN-Prompt使Claude和ChatGPT的零样本性能可与文档图像问答任务上最先进方法的微调性能相媲美，而LATIN-Tuning显著提升了Alpaca的零样本性能。例如，LATIN-Prompt使Claude和ChatGPT在DocVQA上的性能分别提升263%和20%，LATIN-Tuning使Alpaca在DocVQA上的性能提升87.7%。定量与定性分析证明了LATIN-Prompt和LATIN-Tuning的有效性。我们在补充材料中提供代码，并将公开以促进未来研究。