Learning style refers to a type of training mechanism adopted by an individual to gain new knowledge. As suggested by the VARK model, humans have different learning preferences like visual, auditory, etc., for acquiring and effectively processing information. Inspired by this concept, our work explores the idea of mixed information sharing with model compression in the context of Knowledge Distillation (KD) and Mutual Learning (ML). Unlike conventional techniques that share the same type of knowledge with all networks, we propose to train individual networks with different forms of information to enhance the learning process. We formulate a combined KD and ML framework with one teacher and two student networks that share or exchange information in the form of predictions and feature maps. Our comprehensive experiments with benchmark classification and segmentation datasets demonstrate that with 15% compression, the ensemble performance of networks trained with diverse forms of knowledge outperforms the conventional techniques both quantitatively and qualitatively.

翻译：学习风格是指个体为获取新知识所采用的训练机制类型。根据VARK模型的启示，人类具有视觉、听觉等不同学习偏好，用于获取和有效处理信息。受此概念启发，本研究探索了在知识蒸馏（KD）和互学习（ML）框架下混合信息共享与模型压缩的思路。与传统技术向所有网络传递相同类型知识不同，我们提出用不同形式的信息训练个体网络以增强学习过程。我们构建了一个结合KD与ML的框架，包含一个教师网络和两个学生网络，通过网络间预测结果与特征图的共享或交换实现信息传递。基于基准分类与分割数据集的综合实验表明，在15%压缩率条件下，采用多样化知识形式训练的网络集成性能在定量与定性层面均优于传统技术。