Just like the previous generation of task-tuned models, large language models (LLMs) that are adapted to tasks via prompt-based methods like in-context-learning (ICL) perform well in some setups but not in others. This lack of consistency in prompt-based learning hints at a lack of robust generalisation. We here introduce the ICL consistency test -- a contribution to the GenBench collaborative benchmark task (CBT) -- which evaluates how consistent a model makes predictions across many different setups while using the same data. The test is based on different established natural language inference tasks. We provide preprocessed data constituting 96 different 'setups' and a metric that estimates model consistency across these setups. The metric is provided on a fine-grained level to understand what properties of a setup render predictions unstable and on an aggregated level to compare overall model consistency. We conduct an empirical analysis of eight state-of-the-art models, and our consistency metric reveals how all tested LLMs lack robust generalisation.

翻译：正如上一代任务微调模型一样，通过基于提示的方法（如上下文学习）适配任务的大型语言模型在某些设置中表现良好，但在其他设置中则不尽如人意。这种基于提示的学习中缺乏一致性的现象暗示了模型鲁棒泛化能力的不足。本文提出ICL一致性测试——作为GenBench协作基准任务（CBT）的贡献——该测试评估模型在使用相同数据时，在不同设置下预测结果的一致性。该测试基于多个经典的自然语言推理任务。我们提供了经过预处理的包含96种不同"设置"的数据集，以及一个用于估算模型在这些设置间一致性的指标。该指标在细粒度层面可理解何种设置属性导致预测不稳定，在聚合层面则用于比较模型的整体一致性。我们对八种最先进模型进行了实证分析，一致性指标揭示出所有受测的大型语言模型均缺乏鲁棒泛化能力。