A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks. VFMs like CLIP, DINOv2, SAM are trained with distinct objectives, exhibiting unique characteristics for various downstream tasks. We find that despite their conceptual differences, these models can be effectively merged into a unified model through multi-teacher distillation. We name this approach AM-RADIO (Agglomerative Model -- Reduce All Domains Into One). This integrative approach not only surpasses the performance of individual teacher models but also amalgamates their distinctive features, such as zero-shot vision-language comprehension, detailed pixel-level understanding, and open vocabulary segmentation capabilities. In pursuit of the most hardware-efficient backbone, we evaluated numerous architectures in our multi-teacher distillation pipeline using the same training recipe. This led to the development of a novel architecture (E-RADIO) that exceeds the performance of its predecessors and is at least 7x faster than the teacher models. Our comprehensive benchmarking process covers downstream tasks including ImageNet classification, ADE20k semantic segmentation, COCO object detection and LLaVa-1.5 framework. Code: https://github.com/NVlabs/RADIO

翻译：摘要：近期，少数视觉基础模型（VFM）已成为众多下游任务的骨干网络。诸如CLIP、DINOv2、SAM等VFM通过不同目标函数进行训练，展现出各自独特的特性以应对各类下游任务。我们发现，尽管这些模型在概念上存在差异，但通过多教师蒸馏技术，它们可以有效地融合为一个统一模型。我们将此方法命名为AM-RADIO（聚合式模型——融合所有领域于一体）。这种整合方法不仅超越了单个教师模型的性能，还综合了它们各自的独特优势，例如零样本视觉-语言理解、精细像素级理解以及开放词汇分割能力。为追求最高效的硬件骨干架构，我们在多教师蒸馏流程中采用相同的训练方案，对多种架构进行了评估。由此，我们开发出一种新型架构（E-RADIO），其性能超越了先前模型，且速度至少是教师模型的7倍。我们的综合基准测试涵盖了包括ImageNet分类、ADE20k语义分割、COCO目标检测以及LLaVa-1.5框架在内的下游任务。代码链接：https://github.com/NVlabs/RADIO