Recent advancements in foundation models have yielded impressive performance across a wide range of tasks. Meanwhile, for specific applications, practitioners have been developing specialized application models. To enjoy the benefits of both kinds of models, one natural path is to transfer the knowledge in foundation models into specialized application models, which are generally more efficient for serving. Techniques from knowledge distillation may be applied here, where the application model learns to mimic the foundation model. However, specialized application models and foundation models have substantial gaps in capacity, employing distinct architectures, using different input features from different modalities, and being optimized on different distributions. These differences in model characteristics lead to significant challenges for distillation methods. In this work, we propose creating a teaching committee comprising both foundation model teachers and complementary teachers. Complementary teachers possess model characteristics akin to the student's, aiming to bridge the gap between the foundation model and specialized application models for a smoother knowledge transfer. Further, to accommodate the dissimilarity among the teachers in the committee, we introduce DiverseDistill, which allows the student to understand the expertise of each teacher and extract task knowledge. Our evaluations demonstrate that adding complementary teachers enhances student performance. Finally, DiverseDistill consistently outperforms baseline distillation methods, regardless of the teacher choices, resulting in significantly improved student performance.

翻译：近期基础模型的进展在广泛任务上取得了令人瞩目的性能。然而，针对特定应用，从业者一直在开发专用应用模型。为兼得两类模型的优势，一个自然的途径是将基础模型中的知识迁移至通常更高效服务的专用应用模型中。知识蒸馏技术可应用于此，使应用模型学习模仿基础模型。然而，专用应用模型与基础模型在能力上存在显著差距：采用不同架构、使用来自不同模态的输入特征，并在不同分布上进行优化。这些模型特征的差异给蒸馏方法带来了重大挑战。在本工作中，我们提出构建一个由基础模型教师和互补教师共同组成的教学委员会。互补教师拥有与学生模型相似的模型特征，旨在弥合基础模型与专用应用模型之间的鸿沟，实现更平滑的知识迁移。此外，为适应委员会中教师间的异质性，我们引入了DiverseDistill，该方法使学生模型能够理解每位教师的专长并提取任务知识。我们的评估表明，添加互补教师可提升学生性能。最终，无论教师选择如何，DiverseDistill均持续优于基线蒸馏方法，显著改善了学生性能。