During the continuous evolution of one organism's ancestry, its genes accumulate extensive experiences and knowledge, enabling newborn descendants to rapidly adapt to their specific environments. Motivated by this observation, we propose a novel machine learning paradigm \textit{Learngene} to enable learning models to incorporate three key characteristics of genes. (i) Accumulating: the knowledge is accumulated during the continuous learning of an \textbf{ancestry model}. (ii) Condensing: the exhaustive accumulated knowledge is condensed into a much more compact information piece, \ie \textbf{learngene}. (iii): Inheriting: the condensed \textbf{learngene} is inherited to make it easier for \textbf{descendant models} to adapt to new environments. Since accumulating has been studied in some well-developed paradigms like large-scale pre-training and lifelong learning, we focus on condensing and inheriting, which induces three key issues and we provide the preliminary solutions to these issues in this paper: (i) \textit{Learngene} Form: the \textbf{learngene} is set to a few integral layers that can preserve the most commonality. (ii) \textit{Learngene} Condensing: we identify which layers among the ancestry model have the most similarity as one pseudo descendant model. (iii) \textit{Learngene} Inheriting: to construct distinct descendant models for specific downstream tasks, we stack some randomly initialized layers to the \textbf{learngene} layers. Extensive experiments of various settings, including using different network architectures like Vision Transformer (ViT) and Convolutional Neural Networks (CNNs) on different datasets, are carried out to confirm five advantages and two characteristics of \textit{Learngene}.

翻译：摘要：在生物体祖先的持续演化过程中，其基因积累了广泛的经验与知识，使新生后代能够快速适应特定环境。受此启发，我们提出一种新型机器学习范式——\textit{Learngene}，使学习模型具备基因的三个关键特征：（i）积累：知识在\textbf{祖先模型}的持续学习过程中积累；（ii）压缩：将积累的详尽知识压缩成高度紧凑的信息单元，即\textbf{learngene}；（iii）继承：压缩后的\textbf{learngene}被继承，便于\textbf{后代模型}适应新环境。由于积累已在某些成熟范式（如大规模预训练和持续学习）中得到研究，我们重点聚焦压缩与继承，由此引出三个关键问题，并在本文中提供初步解决方案：（i）\textit{Learngene}形式：将\textbf{learngene}设置为能保留最大共性的若干核心层；（ii）\textit{Learngene}压缩：识别祖先模型中与单个伪后代模型最相似的层；（iii）\textit{Learngene}继承：为构建特定下游任务的不同后代模型，在\textbf{learngene}层上堆叠若干随机初始化的层。我们针对不同设置（包括在不同数据集上使用Vision Transformer (ViT)和卷积神经网络(CNNs)等不同网络架构）开展了大量实验，验证了\textit{Learngene}的五项优势与两个特性。