The quality of self-supervised pre-trained embeddings on out-of-distribution (OOD) data is poor without fine-tuning. A straightforward and simple approach to improving the generalization of pre-trained representation to OOD data is the use of deep ensembles. However, obtaining an effective ensemble in the embedding space with only unlabeled data remains an unsolved problem. We first perform a theoretical analysis that reveals the relationship between individual hyperspherical embedding spaces in an ensemble. We then design a principled method to align these embedding spaces in an unsupervised manner. Experimental results on the MNIST dataset show that our embedding-space ensemble method improves pre-trained embedding quality on in-distribution and OOD data compared to single encoders.

翻译：自监督预训练嵌入在未经微调的情况下，对分布外（OOD）数据的表征质量往往较差。一种直接且简单的改进方法是利用深度集成学习来增强预训练表征对OOD数据的泛化能力。然而，仅使用无标注数据在嵌入空间中构建有效的集成模型仍是一个未解决的问题。我们首先通过理论分析揭示了集成中各个超球面嵌入空间之间的内在关联。随后，我们设计了一种基于原理的无监督方法来实现这些嵌入空间的对齐。在MNIST数据集上的实验结果表明，相较于单一编码器，我们的嵌入空间集成方法能够显著提升预训练嵌入在分布内和分布外数据上的表征质量。