Recently, large language models (LLMs) and multimodal large language models (MLLMs) have demonstrated promising results on document visual question answering (VQA) task, particularly after training on document instruction datasets. An effective evaluation method for document instruction data is crucial in constructing instruction data with high efficacy, which, in turn, facilitates the training of LLMs and MLLMs for document VQA. However, most existing evaluation methods for instruction data are limited to the textual content of the instructions themselves, thereby hindering the effective assessment of document instruction datasets and constraining their construction. In this paper, we propose ProcTag, a data-oriented method that assesses the efficacy of document instruction data. ProcTag innovatively performs tagging on the execution process of instructions rather than the instruction text itself. By leveraging the diversity and complexity of these tags to assess the efficacy of the given dataset, ProcTag enables selective sampling or filtering of document instructions. Furthermore, DocLayPrompt, a novel semi-structured layout-aware document prompting strategy, is proposed for effectively representing documents. Experiments demonstrate that sampling existing open-sourced and generated document VQA/instruction datasets with ProcTag significantly outperforms current methods for evaluating instruction data. Impressively, with ProcTag-based sampling in the generated document datasets, only 30.5\% of the document instructions are required to achieve 100\% efficacy compared to the complete dataset. The code is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/ProcTag.

翻译：近年来，大型语言模型（LLMs）与多模态大型语言模型（MLLMs）在文档视觉问答（VQA）任务上展现出令人瞩目的成果，尤其是在经过文档指令数据集训练之后。一种有效的文档指令数据评估方法对于构建高效能的指令数据至关重要，进而有助于训练面向文档VQA的LLMs与MLLMs。然而，现有的大多数指令数据评估方法仅局限于指令本身的文本内容，这阻碍了对文档指令数据集的有效评估，并制约了其构建。本文提出ProcTag，一种面向数据的、用于评估文档指令数据效能的方法。ProcTag创新性地对指令的执行过程而非指令文本本身进行标注。通过利用这些标注的多样性与复杂性来评估给定数据集的效能，ProcTag能够实现对文档指令的选择性采样或过滤。此外，本文提出一种新颖的半结构化布局感知文档提示策略DocLayPrompt，用于有效表征文档。实验表明，使用ProcTag对现有开源及生成的文档VQA/指令数据集进行采样，其性能显著优于当前评估指令数据的方法。值得注意的是，在生成的文档数据集中，基于ProcTag的采样仅需使用30.5%的文档指令即可达到与完整数据集相当的100%效能。代码已公开于https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/ProcTag。