We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we introduce a set of fourteen corruption functions that generate incorrect sentences by systematically introducing errors into existing correct Danish sentences. To ensure the accuracy of these corruptions, we assess their validity using both manual and automatic methods. The results are then used as a benchmark for evaluating Large Language Models on a linguistic acceptability judgement task. Our findings demonstrate that this extension is both broader and more comprehensive than the current state of the art. By incorporating a greater variety of corruption types, our benchmark provides a more rigorous assessment of linguistic acceptability, increasing task difficulty, as evidenced by the lower performance of LLMs on our benchmark compared to existing ones. Our results also suggest that our benchmark has a higher discriminatory power which allows to better distinguish well-performing models from low-performing ones.

翻译：本文提出了一种用于评估丹麦语语言可接受性的增强型基准。我们首先分析了书面丹麦语中最常见的错误类型。基于此分析，我们引入了一套包含十四种破坏性函数的方法，通过系统性地在现有正确丹麦语句子中引入错误来生成错误句子。为确保这些破坏性操作的准确性，我们采用人工与自动方法相结合的方式对其有效性进行了评估。所得结果随后被用作评估大型语言模型在语言可接受性判断任务上的基准。我们的研究表明，该扩展基准在覆盖范围和综合性上均优于当前最优方法。通过纳入更多样化的错误类型，我们的基准提供了更严格的语言可接受性评估，这体现在大型语言模型在本基准上的性能低于现有基准，从而证明了任务难度的提升。实验结果同时表明，本基准具有更高的区分度，能够更有效地区分高性能模型与低性能模型。