Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models -- known as foundation models -- which can be easily adapted to a wide range of downstream tasks. The current paradigm for comparing foundation models involves benchmarking them with aggregate metrics on various curated datasets. Unfortunately, this method of model comparison is heavily dependent on the choice of metric, which makes it unsuitable for situations where the ideal metric is either not obvious or unavailable. In this work, we present a metric-free methodology for comparing foundation models via their embedding space geometry. Our methodology is grounded in random graph theory, and facilitates both pointwise and multi-model comparison. Further, we demonstrate how our framework can be used to induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics.

翻译：近期自监督学习与神经网络规模扩展的进展催生了被称为基础模型的大型模型，这些模型可轻松适配各类下游任务。当前比较基础模型的范式涉及在多种整理数据集上使用聚合指标进行基准测试。遗憾的是，这种模型比较方法高度依赖指标选择，使得在理想指标不明确或不可用的情况下难以适用。本研究提出一种无指标方法，通过基础模型的嵌入空间几何结构进行比较。该方法基于随机图理论，支持逐点比较与多模型比较。此外，我们展示了如何利用该框架构建一个配备距离函数的模型流形，该距离函数与多项下游指标强相关。