During the continuous evolution of one organism's ancestry, its genes accumulate extensive experiences and knowledge, enabling newborn descendants to rapidly adapt to their specific environments. Motivated by this observation, we propose a novel machine learning paradigm Learngene to enable learning models to incorporate three key characteristics of genes. (i) Accumulating: the knowledge is accumulated during the continuous learning of an ancestry model. (ii) Condensing: the extensive accumulated knowledge is condensed into a much more compact information piece, i.e., learngene. (iii) Inheriting: the condensed learngene is inherited to make it easier for descendant models to adapt to new environments. Since accumulating has been studied in well-established paradigms like large-scale pre-training and lifelong learning, we focus on condensing and inheriting, which induces three key issues and we provide the preliminary solutions to these issues in this paper: (i) Learngene Form: the learngene is set to a few integral layers that can preserve significance. (ii) Learngene Condensing: we identify which layers among the ancestry model have the most similarity as one pseudo descendant model. (iii) Learngene Inheriting: to construct distinct descendant models for the specific downstream tasks, we stack some randomly initialized layers to the learngene layers. Extensive experiments across various settings, including using different network architectures like Vision Transformer (ViT) and Convolutional Neural Networks (CNNs) on different datasets, are carried out to confirm four advantages of Learngene: it makes the descendant models 1) converge more quickly, 2) exhibit less sensitivity to hyperparameters, 3) perform better, and 4) require fewer training samples to converge.

翻译：摘要：在生物体祖先的持续进化过程中，其基因积累了丰富的经验和知识，使新生后代能够快速适应特定环境。受此启发，我们提出了一种新型机器学习范式Learngene，使学习模型能够具备基因的三个关键特性：（i）积累：知识在祖先模型的持续学习过程中积累；（ii）浓缩：将积累的广泛知识压缩为更紧凑的信息片段，即learngene；（iii）继承：继承浓缩后的learngene，使后代模型更易适应新环境。由于积累机制已在大规模预训练和终身学习等成熟范式中得到研究，本文聚焦于浓缩与继承环节，由此引出三个关键问题并给出初步解决方案：（i）Learngene形式：将learngene设定为若干能够保持重要性的完整层；（ii）Learngene浓缩：识别祖先模型中与伪后代模型最相似的层；（iii）Learngene继承：为构建适用于特定下游任务的不同后代模型，在learngene层堆叠若干随机初始化的层。通过在不同设置下（包括在多样化数据集上使用Vision Transformer（ViT）和卷积神经网络（CNN）等不同网络架构）进行大量实验，验证了Learngene的四项优势：它能使后代模型1）更快收敛，2）对超参数敏感性更低，3）表现更优，4）收敛所需训练样本更少。