Text editing is a crucial task that involves modifying text to better align with user intents. However, existing text editing benchmark datasets have limitations in providing only coarse-grained instructions. Consequently, although the edited output may seem reasonable, it often deviates from the intended changes outlined in the gold reference, resulting in low evaluation scores. To comprehensively investigate the text editing capabilities of large language models, this paper introduces XATU, the first benchmark specifically designed for fine-grained instruction-based explainable text editing. XATU covers a wide range of topics and text types, incorporating lexical, syntactic, semantic, and knowledge-intensive edits. To enhance interpretability, we leverage high-quality data sources and human annotation, resulting in a benchmark that includes fine-grained instructions and gold-standard edit explanations. By evaluating existing open and closed large language models against our benchmark, we demonstrate the effectiveness of instruction tuning and the impact of underlying architecture across various editing tasks. Furthermore, extensive experimentation reveals the significant role of explanations in fine-tuning language models for text editing tasks. The benchmark will be open-sourced to support reproduction and facilitate future research.

翻译：文本编辑是一项涉及根据用户意图修改文本的关键任务。然而，现有文本编辑基准数据集仅能提供粗粒度指令，这导致编辑结果虽看似合理，却常偏离黄金标准参照中明确的修改目标，造成评估分数偏低。为全面探究大语言模型的文本编辑能力，本文提出XATU——首个专为细粒度指令驱动可解释文本编辑设计的基准。该基准涵盖广泛主题与文本类型，集成词汇、句法、语义及知识密集型编辑操作。为增强可解释性，我们利用高质量数据源与人工标注，构建了包含细粒度指令与标准编辑解释的基准数据集。通过评估现有开源与闭源大语言模型在该基准上的表现，我们验证了指令微调的有效性，以及底层架构对不同编辑任务的影响。此外，大量实验揭示了编辑解释在文本编辑任务语言模型微调中的关键作用。该基准将开源发布，以支持复现并促进未来研究。