An explosion of work in language is leading to ever-increasing numbers of available natural language processing models, with little understanding of how new models compare to better-understood models. One major reason for this difficulty is saturating benchmark datasets, which may not reflect well differences in model performance in the wild. In this work, we propose a novel framework for comparing two natural language processing models by revealing their shared invariance to interpretable input perturbations that are designed to target a specific linguistic capability (e.g., Synonym-Invariance, Typo-Invariance). Via experiments on models from within the same and across different architecture families, this framework offers a number of insights about how changes in models (e.g., distillation, increase in size, amount of pre-training) affect multiple well-defined linguistic capabilities. Furthermore, we also demonstrate how our framework can enable evaluation of the invariances shared between models that are available as commercial black-box APIs (e.g., InstructGPT family) and models that are relatively better understood (e.g., GPT-2). Across several experiments, we observe that large language models share many of the invariances encoded by models of various sizes, whereas the invariances encoded by large language models are only shared by other large models. Possessing a wide variety of invariances may be a key reason for the recent successes of large language models, and our framework can shed light on the types of invariances that are retained by or emerge in new models.

翻译：语言领域的爆发式研究使得可用的自然语言处理模型数量不断增加，但人们对新模型与更成熟的模型之间的比较却知之甚少。造成这一困难的主要原因之一是基准数据集趋于饱和，可能无法充分反映模型在实际应用中的性能差异。本文提出了一种新颖的框架，用于比较两种自然语言处理模型，通过揭示它们对旨在针对特定语言能力（如同义不变性、拼写错误不变性）的可解释输入扰动所共享的不变性。通过对同一架构族内及不同架构族间的模型进行实验，该框架就模型的变化（如蒸馏、规模扩大、预训练量增加）如何影响多个明确定义的语言能力提供了诸多见解。此外，我们还展示了该框架如何能够评估作为商业黑盒API提供的模型（如InstructGPT系列）与相对更易理解的模型（如GPT-2）之间共享的不变性。在多项实验中，我们观察到大型语言模型共享了多种规模模型编码的不变性，而大型语言模型编码的不变性仅由其他大型模型共享。拥有广泛的不变性可能是近期大型语言模型成功的关键原因之一，而我们的框架有助于阐明新模型中保留或涌现的不变性类型。