At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/VanillaNet.

翻译：基础模型的核心哲学是“多即不同”，这在计算机视觉和自然语言处理领域的惊人成功中得到了体现。然而，Transformer模型面临的优化挑战与固有复杂性，要求范式转向简洁性。本研究提出VanillaNet——一种拥抱优雅设计的神经网络架构。通过避免高深度、捷径连接以及自注意力等复杂操作，VanillaNet展现出令人耳目一新的简洁性与卓越性能。其各层均被精心设计为紧凑且直观的结构，并在训练后剪除非线性激活函数以恢复原始架构。VanillaNet突破了固有复杂性的限制，使其成为资源受限环境的理想选择。其易于理解且高度简化的架构为高效部署开辟了新可能。大量实验表明，VanillaNet在性能上与著名深度神经网络及视觉Transformer模型比肩，充分彰显了深度学习中的极简主义力量。VanillaNet这一前瞻性探索具有重塑基础模型格局、挑战现有范式的巨大潜力，为优雅高效的模型设计开辟了新路径。预训练模型与代码已在https://github.com/huawei-noah/VanillaNet及https://gitee.com/mindspore/models/tree/master/research/cv/VanillaNet开源。