Standard evaluation in NLP typically indicates that system A is better on average than system B, but it provides little info on how to improve performance and, what is worse, it should not come as a surprise if B ends up being better than A on outside data. We propose an evaluation methodology for sequence labeling tasks grounded on error analysis that provides both quantitative and qualitative information on where systems must be improved and predicts how models will perform on a different distribution. The key is to create test sets that, contrary to common practice, do not rely on gathering large amounts of real-world in-distribution scraped data, but consists in handcrafting a small set of linguistically motivated examples that exhaustively cover the range of span attributes (such as shape, length, casing, sentence position, etc.) a system may encounter in the wild. We demonstrate this methodology on a benchmark for anglicism identification in Spanish. Our methodology provides results that are diagnostic (because they help identify systematic weaknesses in performance), actionable (because they can inform which model is better suited for a given scenario) and predictive: our method predicts model performance on external datasets with a median correlation of 0.85.

翻译：自然语言处理中的标准评估通常仅表明系统A在平均性能上优于系统B，但几乎无法提供如何改进性能的信息；更严重的是，当面对外部数据时，若系统B最终表现优于系统A，也不应令人感到意外。本文提出一种基于错误分析的序列标注任务评估方法，该方法能同时提供系统需改进之处的定量与定性信息，并预测模型在不同数据分布上的表现。其关键在于构建与常规实践相反的测试集：不依赖于收集大量真实场景中同分布的网络爬取数据，而是通过人工构建少量基于语言学动机的样例，这些样例需详尽覆盖系统在真实场景中可能遇到的所有跨度属性范围（如形态、长度、大小写、句子位置等）。我们在西班牙语中英语借词识别的基准测试中验证了此方法。本方法提供的评估结果具有诊断性（因其有助于识别性能中的系统性缺陷）、可操作性（因其能指导特定场景下更适用模型的选择）及预测性：我们的方法在外部数据集上预测模型性能的中位数相关系数达到0.85。