Do different neural networks, trained for various vision tasks, share some common representations? In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). We present an algorithm for mining a dictionary of Rosetta Neurons across several popular vision models: Class Supervised-ResNet50, DINO-ResNet50, DINO-ViT, MAE, CLIP-ResNet50, BigGAN, StyleGAN-2, StyleGAN-XL. Our findings suggest that certain visual concepts and structures are inherently embedded in the natural world and can be learned by different models regardless of the specific task or architecture, and without the use of semantic labels. We can visualize shared concepts directly due to generative models included in our analysis. The Rosetta Neurons facilitate model-to-model translation enabling various inversion-based manipulations, including cross-class alignments, shifting, zooming, and more, without the need for specialized training.

翻译：不同的神经网络在训练用于各种视觉任务时，是否共享某些共同表示？本文证明，在一系列具有不同架构、不同任务（生成式与判别式）以及不同监督方式（类别监督、文本监督、自监督）的模型中，存在我们称为"Rosetta 神经元"的通用特征。我们提出一种算法，用于挖掘多个主流视觉模型中的 Rosetta 神经元词典，包括：Class Supervised-ResNet50、DINO-ResNet50、DINO-ViT、MAE、CLIP-ResNet50、BigGAN、StyleGAN-2 和 StyleGAN-XL。我们的发现表明，某些视觉概念和结构内在地嵌入于自然世界中，且无论具体任务或架构如何，也无需使用语义标签，不同模型均可学习到这些概念。由于分析中包含了生成模型，我们能够直接可视化这些共享概念。Rosetta 神经元促进了模型间的转换，支持多种基于反转的操作，包括跨类别对齐、平移、缩放等，且无需专门的训练。