In recent years, many NLP studies have focused solely on performance improvement. In this work, we focus on the linguistic and scientific aspects of NLP. We use the task of generating referring expressions in context (REG-in-context) as a case study and start our analysis from GREC, a comprehensive set of shared tasks in English that addressed this topic over a decade ago. We ask what the performance of models would be if we assessed them (1) on more realistic datasets, and (2) using more advanced methods. We test the models using different evaluation metrics and feature selection experiments. We conclude that GREC can no longer be regarded as offering a reliable assessment of models' ability to mimic human reference production, because the results are highly impacted by the choice of corpus and evaluation metrics. Our results also suggest that pre-trained language models are less dependent on the choice of corpus than classic Machine Learning models, and therefore make more robust class predictions.

翻译：近年来，许多自然语言处理研究仅聚焦于性能提升。本文则致力于探究自然语言处理的语言学与科学维度。我们以语境化指称表达生成任务为案例研究，从十多年前针对该主题开展的综合性英语共享任务集GREC入手，分析以下问题：若采用（1）更贴近现实的数据集及（2）更先进的方法进行评估，模型表现将呈现何种变化？我们运用不同评价指标与特征选择实验对模型进行测试。结果表明，由于结果高度依赖于语料库选择与评价指标，GREC已不再能可靠评估模型模拟人类指称生成的能力。此外，我们的发现显示，相较于传统机器学习模型，预训练语言模型对语料库选择的依赖性更弱，因此其类别预测更具鲁棒性。