New knowledge originates from the old. The various types of elements, deposited in the training history, are a large amount of wealth for improving learning deep models. In this survey, we comprehensively review and summarize the topic--``Historical Learning: Learning Models with Learning History'', which learns better neural models with the help of their learning history during its optimization, from three detailed aspects: Historical Type (what), Functional Part (where) and Storage Form (how). To our best knowledge, it is the first survey that systematically studies the methodologies which make use of various historical statistics when training deep neural networks. The discussions with related topics like recurrent/memory networks, ensemble learning, and reinforcement learning are demonstrated. We also expose future challenges of this topic and encourage the community to pay attention to the think of historical learning principles when designing algorithms. The paper list related to historical learning is available at \url{https://github.com/Martinser/Awesome-Historical-Learning.}

翻译：新知源于旧知。训练过程中沉淀的各类历史要素，是改进深度学习模型的宝贵财富。本综述从历史类型（What）、功能模块（Where）与存储形式（How）三个维度，全面梳理与总结了"历史学习：利用学习历史训练深度学习模型"这一主题——即在优化过程中借助学习历史来训练更优的神经网络模型。据我们所知，这是首篇系统研究利用深度学习训练过程中的各类历史统计数据的方法学综述。我们探讨了该主题与循环/记忆网络、集成学习、强化学习等关联领域的交叉关系，并揭示了该领域面临的未来挑战，鼓励学界在设计算法时重视历史学习原则的思考。历史学习相关论文列表见：\url{https://github.com/Martinser/Awesome-Historical-Learning}