Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models, known as foundation models, which can be easily adapted to a wide range of downstream tasks. The current paradigm for comparing foundation models involves evaluating them with aggregate metrics on various benchmark datasets. This method of model comparison is heavily dependent on the chosen evaluation metric, which makes it unsuitable for situations where the ideal metric is either not obvious or unavailable. In this work, we present a methodology for directly comparing the embedding space geometry of foundation models, which facilitates model comparison without the need for an explicit evaluation metric. Our methodology is grounded in random graph theory and enables valid hypothesis testing of embedding similarity on a per-datum basis. Further, we demonstrate how our methodology can be extended to facilitate population level model comparison. In particular, we show how our framework can induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics. We remark on the utility of this population level model comparison as a first step towards a taxonomic science of foundation models.

翻译：近期自监督学习与神经网络规模化的进展催生了被称为基础模型的大型模型，这些模型可轻松适配各类下游任务。当前比较基础模型的范式是在多个基准数据集上通过聚合指标进行评估。这种模型比较方法高度依赖于所选评估指标，导致其不适用于理想指标不明确或不可用的情况。本研究提出一种直接比较基础模型嵌入空间几何特性的方法论，无需显式评估指标即可实现模型比较。该方法基于随机图理论，能够实现逐数据点的嵌入相似性有效假设检验。此外，我们展示了如何将该方法扩展以支持群体层面的模型比较。具体而言，我们证明该框架可生成一个配备距离函数的模型流形，该距离函数与多项下游指标高度相关。我们指出这种群体层面模型比较的实用价值，将其作为迈向基础模型分类科学的初步步骤。