Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is a need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations, it also measures the zero-shot transferability of the uncertainty estimate using a novel metric. We apply URL to evaluate eleven uncertainty quantifiers that are pretrained on ImageNet and transferred to eight downstream datasets. We find that approaches that focus on the uncertainty of the representation itself or estimate the prediction risk directly outperform those that are based on the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that it is not necessarily in conflict with traditional representation learning goals. Code is provided under https://github.com/mkirchhof/url .

翻译：表征学习显著推动了预训练模型的发展，这些模型在迁移至新数据集时可作为有价值的起点。随着对可靠机器学习和不确定性量化需求的日益增长，不仅需要能提供嵌入表示的预训练模型，还需要能提供可迁移不确定性估计的模型。为引导此类模型的开发，我们提出了不确定性感知表征学习（URL）基准。除表征的可迁移性外，该基准还采用一种新颖的指标评估不确定性估计的零样本可迁移性。我们应用URL评估了十一种在ImageNet上预训练并迁移至八个下游数据集的不确定性量化方法。研究发现，关注表征自身不确定性或直接估计预测风险的方法，其表现优于基于上游类别概率的方法。然而，实现可迁移的不确定性量化仍是一个开放挑战。我们的结果表明，这并不一定与传统表征学习目标相冲突。代码见 https://github.com/mkirchhof/url 。