Recently, large language models (LLMs) and multimodal large language models (MLLMs) have demonstrated promising results on document visual question answering (VQA) task, particularly after training on document instruction datasets. An effective evaluation method for document instruction data is crucial in constructing instruction data with high efficacy, which, in turn, facilitates the training of LLMs and MLLMs for document VQA. However, most existing evaluation methods for instruction data are limited to the textual content of the instructions themselves, thereby hindering the effective assessment of document instruction datasets and constraining their construction. In this paper, we propose ProcTag, a data-oriented method that assesses the efficacy of document instruction data. ProcTag innovatively performs tagging on the execution process of instructions rather than the instruction text itself. By leveraging the diversity and complexity of these tags to assess the efficacy of the given dataset, ProcTag enables selective sampling or filtering of document instructions. Furthermore, DocLayPrompt, a novel semi-structured layout-aware document prompting strategy, is proposed for effectively representing documents. Experiments demonstrate that sampling existing open-sourced and generated document VQA/instruction datasets with ProcTag significantly outperforms current methods for evaluating instruction data. Impressively, with ProcTag-based sampling in the generated document datasets, only 30.5\% of the document instructions are required to achieve 100\% efficacy compared to the complete dataset. The code is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/ProcTag.

翻译：近期，大型语言模型（LLMs）与多模态大型语言模型（MLLMs）在文档视觉问答（VQA）任务上展现出显著潜力，尤其是在经过文档指令数据集训练后。构建高效的文档指令数据需要有效的评估方法，而高质量的指令数据又能进一步促进面向文档VQA的LLMs与MLLMs训练。然而，现有的大多数指令数据评估方法仅局限于指令本身的文本内容，这阻碍了对文档指令数据集的有效评估，并制约了其构建。本文提出ProcTag，一种面向数据的文档指令数据效能评估方法。ProcTag的创新之处在于对指令执行过程而非指令文本本身进行标注。通过利用这些标注的多样性与复杂性来评估给定数据集的效能，ProcTag能够实现对文档指令的选择性采样或过滤。此外，本文还提出了DocLayPrompt，一种新颖的半结构化布局感知文档提示策略，用于有效表征文档。实验表明，使用ProcTag对现有开源及生成的文档VQA/指令数据集进行采样，其效果显著优于当前主流的指令数据评估方法。值得注意的是，在生成的文档数据集中，基于ProcTag的采样仅需使用30.5%的文档指令即可达到与完整数据集相当的100%效能。代码已公开于https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/ProcTag。