Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is a need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations, it also measures the zero-shot transferability of the uncertainty estimate using a novel metric. We apply URL to evaluate eleven uncertainty quantifiers that are pretrained on ImageNet and transferred to eight downstream datasets. We find that approaches that focus on the uncertainty of the representation itself or estimate the prediction risk directly outperform those that are based on the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that it is not necessarily in conflict with traditional representation learning goals. Code is provided under https://github.com/mkirchhof/url .

翻译：表征学习显著推动了领域发展，使得预训练模型能够在新数据集迁移时作为有价值的起点。随着对可靠机器学习和不确定性量化的需求日益增长，需要不仅能提供嵌入表示、还能提供可迁移不确定性估计的预训练模型。为引导此类模型的开发，我们提出了不确定性感知表征学习（URL）基准。该基准除考量表征的可迁移性外，还通过新颖的评估指标衡量不确定性估计的零样本迁移能力。我们应用URL对11个在ImageNet上预训练并在8个下游数据集迁移的不确定性量化方法进行评估。研究发现，专注于表征本身不确定性或直接估计预测风险的方法，优于基于上游类别概率的方法。然而，实现可迁移的不确定性量化仍是一项开放挑战。研究结果表明，这一目标与传统表征学习目标未必冲突。代码见https://github.com/mkirchhof/url 。