During the continuous evolution of one organism's ancestry, its genes accumulate extensive experiences and knowledge, enabling newborn descendants to rapidly adapt to their specific environments. Motivated by this observation, we propose a novel machine learning paradigm Learngene to enable learning models to incorporate three key characteristics of genes. (i) Accumulating: the knowledge is accumulated during the continuous learning of an ancestry model. (ii) Condensing: the extensive accumulated knowledge is condensed into a much more compact information piece, i.e., learngene. (iii) Inheriting: the condensed learngene is inherited to make it easier for descendant models to adapt to new environments. Since accumulating has been studied in well-established paradigms like large-scale pre-training and lifelong learning, we focus on condensing and inheriting, which induces three key issues and we provide the preliminary solutions to these issues in this paper: (i) Learngene Form: the learngene is set to a few integral layers that can preserve significance. (ii) Learngene Condensing: we identify which layers among the ancestry model have the most similarity as one pseudo descendant model. (iii) Learngene Inheriting: to construct distinct descendant models for the specific downstream tasks, we stack some randomly initialized layers to the learngene layers. Extensive experiments across various settings, including using different network architectures like Vision Transformer (ViT) and Convolutional Neural Networks (CNNs) on different datasets, are carried out to confirm four advantages of Learngene: it makes the descendant models 1) converge more quickly, 2) exhibit less sensitivity to hyperparameters, 3) perform better, and 4) require fewer training samples to converge.

翻译：在生物体祖先的持续进化过程中，其基因积累了丰富的经验与知识，使得新生后代能够快速适应特定环境。受此现象启发，我们提出一种新型机器学习范式学习基因（Learngene），使学习模型具备基因的三个关键特征：（i）积累：在祖先模型的持续学习过程中积累知识；（ii）浓缩：将积累的广泛知识压缩为更紧凑的信息单元，即学习基因；（iii）继承：将浓缩后的学习基因传递给后代模型，使其更容易适应新环境。由于积累已在成熟范式（如大规模预训练和终身学习）中得到研究，我们重点聚焦浓缩与继承环节，由此引出三个关键问题并在本文中提出初步解决方案：（i）学习基因形式：将学习基因设定为若干能保留重要性的整体层；（ii）学习基因浓缩：识别祖先模型中与伪后代模型相似度最高的若干层；（iii）学习基因继承：为构建面向特定下游任务的不同后代模型，在基因层上堆叠随机初始化的层。我们在不同设置下开展广泛实验，包括在不同数据集上使用Vision Transformer（ViT）和卷积神经网络（CNN）等多种网络架构，验证了学习基因的四项优势：使后代模型1）收敛更快，2）对超参数敏感性更低，3）性能更优，4）收敛所需训练样本更少。