Collaborative learning has emerged as a key paradigm in large-scale intelligent systems, enabling distributed agents to cooperatively train their models while addressing their privacy concerns. Central to this paradigm is knowledge distillation (KD), a technique that facilitates efficient knowledge transfer among agents. However, the underlying mechanisms by which KD leverages memory and knowledge across agents remain underexplored. This paper aims to bridge this gap by offering a comprehensive review of KD in collaborative learning, with a focus on the roles of memory and knowledge. We define and categorize memory and knowledge within the KD process and explore their interrelationships, providing a clear understanding of how knowledge is extracted, stored, and shared in collaborative settings. We examine various collaborative learning patterns, including distributed, hierarchical, and decentralized structures, and provide insights into how memory and knowledge dynamics shape the effectiveness of KD in collaborative learning. Particularly, we emphasize task heterogeneity in distributed learning pattern covering federated learning (FL), multi-agent domain adaptation (MADA), federated multi-modal learning (FML), federated continual learning (FCL), federated multi-task learning (FMTL), and federated graph knowledge embedding (FKGE). Additionally, we highlight model heterogeneity, data heterogeneity, resource heterogeneity, and privacy concerns of these tasks. Our analysis categorizes existing work based on how they handle memory and knowledge. Finally, we discuss existing challenges and propose future directions for advancing KD techniques in the context of collaborative learning.

翻译：协作学习已成为大规模智能系统中的关键范式，使得分布式智能体能够在解决隐私顾虑的同时协同训练模型。该范式的核心是知识蒸馏（KD），这项技术促进了智能体间的高效知识迁移。然而，关于KD如何利用跨智能体的记忆与知识的内在机制仍缺乏深入探索。本文旨在通过全面综述协作学习中的KD来弥合这一空白，重点关注记忆与知识的作用。我们定义并分类了KD过程中的记忆与知识，并探讨了它们之间的相互关系，从而清晰地阐明了在协作环境中知识如何被提取、存储与共享。我们考察了多种协作学习模式，包括分布式、分层式与去中心化结构，并深入分析了记忆与知识动态如何塑造KD在协作学习中的有效性。特别地，我们强调了分布式学习模式中的任务异质性，涵盖联邦学习（FL）、多智能体领域自适应（MADA）、联邦多模态学习（FML）、联邦持续学习（FCL）、联邦多任务学习（FMTL）以及联邦图知识嵌入（FKGE）。此外，我们还重点分析了这些任务中的模型异质性、数据异质性、资源异质性及隐私问题。我们的分析基于现有工作如何处理记忆与知识对其进行了分类。最后，我们讨论了现有挑战，并为在协作学习背景下推进KD技术提出了未来研究方向。