Large vision-language models have achieved outstanding performance, but their size and computational requirements make their deployment on resource-constrained devices and time-sensitive tasks impractical. Model distillation, the process of creating smaller, faster models that maintain the performance of larger models, is a promising direction towards the solution. This paper investigates the distillation of visual representations in large teacher vision-language models into lightweight student models using a small- or mid-scale dataset. Notably, this study focuses on open-vocabulary out-of-distribution (OOD) generalization, a challenging problem that has been overlooked in previous model distillation literature. We propose two principles from vision and language modality perspectives to enhance student's OOD generalization: (1) by better imitating teacher's visual representation space, and carefully promoting better coherence in vision-language alignment with the teacher; (2) by enriching the teacher's language representations with informative and finegrained semantic attributes to effectively distinguish between different labels. We propose several metrics and conduct extensive experiments to investigate their techniques. The results demonstrate significant improvements in zero-shot and few-shot student performance on open-vocabulary out-of-distribution classification, highlighting the effectiveness of our proposed approaches. Code released at https://github.com/xuanlinli17/large_vlm_distillation_ood

翻译：大规模视觉语言模型展现了卓越的性能，但其庞大的规模和计算需求使其在资源受限设备及时间敏感型任务上的部署不切实际。模型蒸馏——通过创建更小、更快的模型来保留大型模型性能的过程——是解决上述问题的一个有前景方向。本文研究如何利用中小规模数据集，将大型教师视觉语言模型中的视觉表征蒸馏至轻量级学生模型中。值得注意的是，本研究聚焦于开放词汇分布外（OOD）泛化这一在以往模型蒸馏文献中被忽视的挑战性问题。我们从视觉与语言模态视角提出两个增强学生模型OOD泛化能力的原则：（1）通过更好地模仿教师的视觉表征空间，并谨慎促进视觉-语言对齐与教师的一致性；（2）通过利用信息丰富且细粒度的语义属性丰富教师的语言表征，以有效区分不同标签。我们提出了多项评估指标并开展了大量实验来探究相关技术。结果表明，在开放词汇分布外分类任务中，学生模型在零样本和少样本场景下的性能得到了显著提升，验证了所提方法的有效性。代码已发布在https://github.com/xuanlinli17/large_vlm_distillation_ood