Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models, known as foundation models, which can be easily adapted to a wide range of downstream tasks. The current paradigm for comparing foundation models involves evaluating them with aggregate metrics on various benchmark datasets. This method of model comparison is heavily dependent on the chosen evaluation metric, which makes it unsuitable for situations where the ideal metric is either not obvious or unavailable. In this work, we present a methodology for directly comparing the embedding space geometry of foundation models, which facilitates model comparison without the need for an explicit evaluation metric. Our methodology is grounded in random graph theory and enables valid hypothesis testing of embedding similarity on a per-datum basis. Further, we demonstrate how our methodology can be extended to facilitate population level model comparison. In particular, we show how our framework can induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics. We remark on the utility of this population level model comparison as a first step towards a taxonomic science of foundation models.

翻译：自监督学习和神经网络规模化的最新进展使得创建大型模型（称为基础模型）成为可能，这些模型可以轻松适应各种下游任务。当前比较基础模型的范式涉及在多个基准数据集上使用聚合指标进行评估。这种模型比较方法高度依赖于所选评估指标，这使得其在理想指标不明确或不可用的情况下不适用。在本文中，我们提出了一种直接比较基础模型嵌入空间几何结构的方法，该方法无需显式评估指标即可实现模型比较。我们的方法基于随机图理论，并能够在逐数据点的基础上对嵌入相似性进行有效的假设检验。此外，我们展示了如何将我们的方法扩展到群体层面的模型比较。特别地，我们说明了我们的框架如何构建一个配备距离函数的模型流形，该距离函数与多个下游指标高度相关。我们指出，这种群体层面模型比较作为迈向基础模型分类科学的第一步具有实用价值。