This paper reexamines the research on out-of-distribution (OOD) robustness in the field of NLP. We find that the distribution shift settings in previous studies commonly lack adequate challenges, hindering the accurate evaluation of OOD robustness. To address these issues, we propose a benchmark construction protocol that ensures clear differentiation and challenging distribution shifts. Then we introduce BOSS, a Benchmark suite for Out-of-distribution robustneSS evaluation covering 5 tasks and 20 datasets. Based on BOSS, we conduct a series of experiments on pre-trained language models for analysis and evaluation of OOD robustness. First, for vanilla fine-tuning, we examine the relationship between in-distribution (ID) and OOD performance. We identify three typical types that unveil the inner learning mechanism, which could potentially facilitate the forecasting of OOD robustness, correlating with the advancements on ID datasets. Then, we evaluate 5 classic methods on BOSS and find that, despite exhibiting some effectiveness in specific cases, they do not offer significant improvement compared to vanilla fine-tuning. Further, we evaluate 5 LLMs with various adaptation paradigms and find that when sufficient ID data is available, fine-tuning domain-specific models outperform LLMs on ID examples significantly. However, in the case of OOD instances, prioritizing LLMs with in-context learning yields better results. We identify that both fine-tuned small models and LLMs face challenges in effectively addressing downstream tasks. The code is public at \url{https://github.com/lifan-yuan/OOD_NLP}.

翻译：本文重新审视了自然语言处理（NLP）领域中关于分布外（OOD）鲁棒性的研究。我们发现，以往研究中的分布偏移设置普遍缺乏足够的挑战性，这阻碍了对OOD鲁棒性的准确评估。为解决这些问题，我们提出了一种基准构建协议，确保明确区分且具有挑战性的分布偏移。随后，我们引入了BOSS（面向分布外鲁棒性评估的基准套件），涵盖5个任务和20个数据集。基于BOSS，我们开展了一系列针对预训练语言模型的实验，用于分析和评估OOD鲁棒性。首先，针对标准微调，我们研究了分布内（ID）与OOD性能之间的关系。我们识别出三种典型类型，揭示了内在学习机制，这有望促进OOD鲁棒性的预测，并与ID数据集的进展相关联。其次，我们在BOSS上评估了5种经典方法，发现尽管这些方法在特定情况下展现出一定效果，但相较于标准微调并未带来显著改进。进一步地，我们评估了采用不同适应范式的5种大语言模型（LLM），发现当有充足的ID数据时，微调领域特定模型在ID示例上的表现显著优于LLM。然而在处理OOD示例时，优先采用基于上下文学习的LLM则能获得更优结果。我们观察到，无论是微调的小型模型还是LLM，在有效处理下游任务时均面临挑战。相关代码已开源至\url{https://github.com/lifan-yuan/OOD_NLP}。