Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models as even subtle changes could incur significant performance drops. Being able to estimate a model's performance on test data is important in practice as it indicates when to trust to model's decisions. We present a simple yet effective method to predict a model's performance on an unknown distribution without any addition annotation. Our approach is rooted in the Optimal Transport theory, viewing test samples' output softmax scores from deep neural networks as empirical samples from an unknown distribution. We show that our method, Confidence Optimal Transport (COT), provides robust estimates of a model's performance on a target domain. Despite its simplicity, our method achieves state-of-the-art results on three benchmark datasets and outperforms existing methods by a large margin.

翻译：分布外（OOD）数据对已部署的机器学习模型构成严重挑战，因为即使是细微的变化也可能导致性能显著下降。能够在测试数据上估计模型性能在实际应用中至关重要，因为它指示了何时可以信任模型的决策。我们提出了一种简单而有效的方法，无需任何额外标注即可预测模型在未知分布上的性能。该方法植根于最优传输理论，将深度神经网络输出的测试样本软最大化分数视为来自未知分布的经验样本。我们证明，我们的方法——置信度最优传输（COT）能够为目标域上的模型性能提供稳健的估计。尽管方法简洁，但我们在三个基准数据集上取得了最先进的结果，并以较大幅度超越了现有方法。