Reviewing plays an important role when learning knowledge. The knowledge acquisition at a certain time point may be strongly inspired with the help of previous experience. Thus the knowledge growing procedure should show strong relationship along the temporal dimension. In our research, we find that during the network training, the evolution of feature map follows temporal sequence property. A proper temporal supervision may further improve the network training performance. Inspired by this observation, we design a novel knowledge distillation method. Specifically, we extract the spatiotemporal features in the different training phases of student by convolutional Long Short-term memory network (Conv-LSTM). Then, we train the student net through a dynamic target, rather than static teacher network features. This process realizes the refinement of old knowledge in student network, and utilizes them to assist current learning. Extensive experiments verify the effectiveness and advantages of our method over existing knowledge distillation methods, including various network architectures, different tasks (image classification and object detection) .

翻译：回顾在学习知识过程中扮演着重要角色。某一时间点的知识获取，往往会从先前的经验中获得强烈启发。因此，知识增长过程应沿着时间维度展现出紧密的关联性。本研究发现，在网络训练过程中，特征图的演化遵循时间序列特性。合理的时序监督可进一步提升网络训练性能。受此观察启发，我们设计了一种新型的知识蒸馏方法。具体而言，我们利用卷积长短期记忆网络（Conv-LSTM）提取学生网络不同训练阶段的时空特征，进而通过动态目标（而非静态教师网络特征）来训练学生网络。该过程实现了学生网络中旧知识的精炼，并利用这些知识辅助当前学习。大量实验验证了本方法相较于现有知识蒸馏方法的有效性与优越性，涵盖多种网络架构及不同任务（图像分类与目标检测）。