Transfer learning and knowledge distillation has recently gained a lot of attention in the deep learning community. One transfer approach, the student-teacher learning, has been shown to successfully create ``small'' student neural networks that mimic the performance of a much bigger and more complex ``teacher'' networks. In this paper, we investigate an extension to this approach and transfer from a non-neural-based machine learning pipeline as teacher to a neural network (NN) student, which would allow for joint optimization of the various pipeline components and a single unified inference engine for multiple ML tasks. In particular, we explore replacing the random forest classifier by transfer learning to a student NN. We experimented with various NN topologies on 100 OpenML tasks in which random forest has been one of the best solutions. Our results show that for the majority of the tasks, the student NN can indeed mimic the teacher if one can select the right NN hyper-parameters. We also investigated the use of random forest for selecting the right NN hyper-parameters.

翻译：迁移学习和知识蒸馏近期在深度学习领域引起了广泛关注。其中一种迁移方法——师生学习机制，已被证明能够成功创建“小型”学生神经网络，使其模仿规模更大、结构更复杂的“教师”网络的性能。本文研究将该方法扩展至非神经网络的机器学习流水线向学生神经网络的迁移，从而实现流水线各组件的联合优化，并为多种机器学习任务提供统一的推理引擎。具体而言，我们探索了通过迁移学习将随机森林分类器替换为学生神经网络的方法。在100个随机森林表现最优的OpenML任务中，我们测试了多种神经网络拓扑结构。结果表明，在大多数任务中，若能选择恰当的神经网络超参数，学生神经网络确实能有效模仿教师网络的表现。此外，我们还研究了利用随机森林选择最佳神经网络超参数的可行性。