Large vision-language models have achieved outstanding performance, but their size and computational requirements make their deployment on resource-constrained devices and time-sensitive tasks impractical. Model distillation, the process of creating smaller, faster models that maintain the performance of larger models, is a promising direction towards the solution. This paper investigates the distillation of visual representations in large teacher vision-language models into lightweight student models using a small- or mid-scale dataset. Notably, this study focuses on open-vocabulary out-of-distribution (OOD) generalization, a challenging problem that has been overlooked in previous model distillation literature. We propose two principles from vision and language modality perspectives to enhance student's OOD generalization: (1) by better imitating teacher's visual representation space, and carefully promoting better coherence in vision-language alignment with the teacher; (2) by enriching the teacher's language representations with informative and finegrained semantic attributes to effectively distinguish between different labels. We propose several metrics and conduct extensive experiments to investigate their techniques. The results demonstrate significant improvements in zero-shot and few-shot student performance on open-vocabulary out-of-distribution classification, highlighting the effectiveness of our proposed approaches. Our code will be released at https://github.com/xuanlinli17/large_vlm_distillation_ood

翻译：大规模视觉-语言模型取得了卓越的性能，但其规模和计算需求使其难以部署在资源受限设备和时间敏感型任务上。模型蒸馏——通过创建更小、更快的模型来保持大型模型性能的过程——是解决这一问题的有前景方向。本文研究如何利用小型或中型数据集，将大规模教师视觉-语言模型中的视觉表示蒸馏到轻量级学生模型中。值得注意的是，本研究聚焦于开放词汇的分布外（OOD）泛化问题，这是以往模型蒸馏文献中未受重视的挑战性课题。我们分别从视觉和语言模态角度提出两项原则来增强学生的OOD泛化能力：（1）通过更好地模仿教师的视觉表示空间，并精心促进与学生间视觉-语言对齐的一致性；（2）通过为教师语言表示补充信息丰富且细粒度的语义属性，以有效区分不同标签。我们提出了多项评估指标并开展了大量实验来研究相关技术。结果表明，在开放词汇分布外分类任务中，学生模型在零样本和少样本场景下的性能均有显著提升，凸显了所提方法的有效性。我们的代码将在https://github.com/xuanlinli17/large_vlm_distillation_ood 开源。