We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data scale and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 5.6M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Finally, we show that task or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. These models required over 10,000 GPU-hours to train and can be found on our website for the benefit of the research community.

翻译：我们呈现了迄今为止规模最大、最全面的关于面向具身AI的预训练视觉表征（PVRs）或视觉基础模型的实证研究。首先，我们构建了包含运动控制、导航、灵巧操作和移动操纵等17项不同任务的CortexBench基准集。随后，我们系统评估了现有PVR模型，发现没有任何模型具有普适性优势。为探究预训练数据规模与多样性的影响，我们整合了来自7个不同来源的超过4000小时第一人称视频（逾560万张图像）与ImageNet数据集，利用掩码自编码（MAE）方法在数据切片上训练不同尺寸的视觉Transformer。与先前研究推论相反，我们发现扩大数据集规模与多样性并未带来普适性的性能提升（但具有平均增益）。我们规模最大的模型VC-1在平均性能上超越了所有先前PVR，但仍未实现普适性主导。最后研究表明，针对特定任务或领域对VC-1进行微调可带来显著提升，经过适配的VC-1在CortexBench所有基准测试中均达到与已知最优结果相当甚至更优的性能。这些模型共消耗超10000GPU小时的训练算力，现已在我们的网站上公开，供科研社区使用。