Do different neural networks, trained for various vision tasks, share some common representations? In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). We present an algorithm for mining a dictionary of Rosetta Neurons across several popular vision models: Class Supervised-ResNet50, DINO-ResNet50, DINO-ViT, MAE, CLIP-ResNet50, BigGAN, StyleGAN-2, StyleGAN-XL. Our findings suggest that certain visual concepts and structures are inherently embedded in the natural world and can be learned by different models regardless of the specific task or architecture, and without the use of semantic labels. We can visualize shared concepts directly due to generative models included in our analysis. The Rosetta Neurons facilitate model-to-model translation enabling various inversion-based manipulations, including cross-class alignments, shifting, zooming, and more, without the need for specialized training.

翻译：不同的神经网络经过训练以完成各种视觉任务后，是否共享某些共同表示？本文证明了我们称为“罗塞塔神经元”的共同特征存在于一系列具有不同架构、不同任务（生成式与判别式）以及不同监督类型（类别监督、文本监督、自监督）的模型中。我们提出了一种挖掘方法，可从多个主流视觉模型中提取罗塞塔神经元词典，这些模型包括：类别监督的ResNet50、DINO-ResNet50、DINO-ViT、MAE、CLIP-ResNet50、BigGAN、StyleGAN-2和StyleGAN-XL。研究结果表明，某些视觉概念与结构天然地嵌入于自然世界中，且无论具体任务或架构如何，不同模型均能习得这些概念，而无需借助语义标签。由于我们的分析纳入了生成模型，因此可直接可视化共享概念。罗塞塔神经元促进了模型间的转换，支持各种基于反演的操作，包括跨类别对齐、位移、缩放等，且无需专门的训练。