Modeling discourse -- the linguistic phenomena that go beyond individual sentences, is a fundamental yet challenging aspect of natural language processing (NLP). However, existing evaluation benchmarks primarily focus on the evaluation of inter-sentence properties and overlook critical discourse phenomena that cross sentences. To bridge the gap, we propose Disco-Bench, a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks, covering understanding, translation, and generation. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena (e.g. cohesion and coherence) in Chinese and/or English. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge. We totally evaluate 20 general-, in-domain and commercial models based on Transformer, advanced pretraining architectures and large language models (LLMs). Our results show (1) the challenge and necessity of our evaluation benchmark; (2) fine-grained pretraining based on literary document-level training data consistently improves the modeling of discourse information. We will release the datasets, pretrained models, and leaderboard, which we hope can significantly facilitate research in this field: https://github.com/longyuewangdcu/Disco-Bench.

翻译：建模话语——即超越单句的语言现象，是自然语言处理中基础且具有挑战性的任务。然而，现有评估基准主要聚焦于句子间属性的评估，忽视了跨句的关键话语现象。为弥补这一空白，我们提出Disco-Bench，该基准能够在涵盖理解、翻译与生成的多类自然语言处理任务中评估句子内话语属性。Disco-Bench包含文学领域的9个文档级测试集，其中蕴含丰富的话语现象（如衔接与连贯），涉及中文和/或英文。为进行语言学分析，我们还设计了一套诊断性测试套件，用于检验目标模型是否学习到话语知识。我们全面评估了20个基于Transformer、先进预训练架构及大语言模型（LLM）的通用型、领域专用型及商业模型。实验结果表明：（1）本评估基准具有挑战性与必要性；（2）基于文学文档级训练数据的细粒度预训练能持续提升话语信息建模能力。我们将公开发布数据集、预训练模型与排行榜，以期显著推动该领域研究：https://github.com/longyuewangdcu/Disco-Bench。