Evaluating the performance of machine learning models under distribution shift is challenging, especially when we only have unlabeled data from the shifted (target) domain, along with labeled data from the original (source) domain. Recent work suggests that the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, is a key to tackle this problem. Experimentally, disagreement and prediction error have been shown to be strongly connected, which has been used to estimate model performance. Experiments have lead to the discovery of the disagreement-on-the-line phenomenon, whereby the classification error under the target domain is often a linear function of the classification error under the source domain; and whenever this property holds, disagreement under the source and target domain follow the same linear relation. In this work, we develop a theoretical foundation for analyzing disagreement in high-dimensional random features regression; and study under what conditions the disagreement-on-the-line phenomenon occurs in our setting. Experiments on CIFAR-10-C, Tiny ImageNet-C, and Camelyon17 are consistent with our theory and support the universality of the theoretical findings.

翻译：评估机器学习模型在分布偏移下的性能具有挑战性，尤其当我们仅拥有偏移（目标）域的无标签数据以及原始（源）域的有标签数据时。近期研究表明，分歧（即采用不同随机性训练的两个模型对同一输入产生的差异程度）是解决该问题的关键。实验证明分歧与预测误差之间存在强关联性，并已被用于估计模型性能。实验发现了分歧在线现象：目标域下的分类误差通常是源域下分类误差的线性函数；且当该性质成立时，源域与目标域下的分歧遵循相同的线性关系。本文为高维随机特征回归中的分歧分析建立了理论基础，并探究了该设置下分歧在线现象成立的条件。在CIFAR-10-C、Tiny ImageNet-C和Camelyon17数据集上的实验结果与理论分析一致，支持了理论发现的普适性。