Machine learning models are increasingly used in practice. However, many machine learning methods are sensitive to test or operational data that is dissimilar to training data. Out-of-distribution (OOD) data is known to increase the probability of error and research into metrics that identify what dissimilarities in data affect model performance is on-going. Recently, combinatorial coverage metrics have been explored in the literature as an alternative to distribution-based metrics. Results show that coverage metrics can correlate with classification error. However, other results show that the utility of coverage metrics is highly dataset-dependent. In this paper, we show that this dataset-dependence can be alleviated with metric learning, a machine learning technique for learning latent spaces where data from different classes is further apart. In a study of 6 open-source datasets, we find that metric learning increased the difference between set-difference coverage metrics (SDCCMs) calculated on correctly and incorrectly classified data, thereby demonstrating that metric learning improves the ability of SDCCMs to anticipate classification error. Paired t-tests validate the statistical significance of our findings. Overall, we conclude that metric learning improves the ability of coverage metrics to anticipate classifier error and identify when OOD data is likely to degrade model performance.

翻译：机器学习模型在实践中日益普及，但许多机器学习方法对与训练数据分布不同的测试数据或运行数据较为敏感。已知分布外数据会增加错误概率，当前研究正持续探索识别影响模型性能的数据差异性的度量指标。近期文献中，组合覆盖度量作为基于分布度量的替代方案得到研究，结果表明覆盖度量能与分类错误产生关联。然而，另有研究发现覆盖度量的有效性高度依赖数据集。本文证明，通过度量学习（一种学习不同类别数据在潜在空间中距离更远的机器学习技术）可缓解这种数据集依赖性。基于6个开源数据集的实验表明，度量学习增大了正确分类数据与错误分类数据在集合差异覆盖度量上的差异，从而验证了度量学习能提升集合差异覆盖度量预测分类错误的能力。配对t检验证实了研究结果的统计显著性。总体而言，我们得出结论：度量学习能增强覆盖度量预测分类器错误的能力，并有效识别可能降低模型性能的分布外数据。