Larger language models have higher accuracy on average, but are they better on every single instance (datapoint)? Some work suggests larger models have higher out-of-distribution robustness, while other work suggests they have lower accuracy on rare subgroups. To understand these differences, we investigate these models at the level of individual instances. However, one major challenge is that individual predictions are highly sensitive to noise in the randomness in training. We develop statistically rigorous methods to address this, and after accounting for pretraining and finetuning noise, we find that our BERT-Large is worse than BERT-Mini on at least 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%. We also find that finetuning noise increases with model size and that instance-level accuracy has momentum: improvement from BERT-Mini to BERT-Medium correlates with improvement from BERT-Medium to BERT-Large. Our findings suggest that instance-level predictions provide a rich source of information; we therefore, recommend that researchers supplement model weights with model predictions.

翻译：较大的语言模型平均具有较高的准确性,但在每个实例(数据点)中,它们是否更好?有些工作表明,较大的模型在分配上具有较高的稳健性,而另一些工作则表明,在稀有分组中,这些模型的准确性较低。为了理解这些差异,我们在个别实例中对这些模型进行了调查;然而,一个重大挑战是,单个预测对随机性中的噪音非常敏感。我们制定了在统计上严格的方法来解决这个问题,在计算了培训前和微调噪音之后,我们发现我们的BERT-Large在MNLI、SST-2和QP至少1-4%的案例中比BERT-Mini要差,而总体精确度则提高2-10%。我们还发现,微调噪音与模型大小相比,这种实例级的准确性具有动力:从BERT-Mini到BERT-Medium的改进与BERT-Mium的改进有关。我们发现,实例级预测提供了丰富的信息来源;因此,我们建议研究人员用模型预测补充模型重量。