Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive zero-shot recognition capability, but still underperform in dense prediction tasks. Self-distillation recently is emerging as a promising approach for fine-tuning VLMs to better adapt to local regions without requiring extensive annotations. However, previous state-of-the-art approaches often suffer from significant `foreground bias', where models tend to wrongly identify background regions as foreground objects. To alleviate this issue, we propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pre-trained VLM representations. To alleviate this issue, we propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pre-trained VLM representations. DenseVLM leverages the pre-trained VLM to retrieve categories for unlabeled regions and then decouples the interference between foreground and background features. We show that DenseVLM can directly replace the original VLM in open-vocabulary object detection and image segmentation methods, leading to notable performance improvements. Furthermore, it exhibits promising zero-shot scalability when training on more extensive and diverse datasets. Our code is available at https://github.com/HVision-NKU/DenseVLM.

翻译：预训练的视觉-语言模型（VLM），例如CLIP，已展现出令人印象深刻的零样本识别能力，但在密集预测任务中仍表现欠佳。自蒸馏技术近期作为一种有前景的方法崭露头角，它能在无需大量标注的情况下微调VLM，使其更好地适应局部区域。然而，先前最先进的方法通常存在严重的“前景偏差”，即模型倾向于错误地将背景区域识别为前景物体。为缓解此问题，我们提出了DenseVLM，一个旨在从强大的预训练VLM表征中学习无偏区域-语言对齐的框架。DenseVLM利用预训练的VLM为未标注区域检索类别，进而解耦前景与背景特征之间的干扰。我们证明，DenseVLM可以直接替代开放词汇目标检测和图像分割方法中的原始VLM，从而带来显著的性能提升。此外，当在更广泛和多样化的数据集上进行训练时，它展现出良好的零样本可扩展性。我们的代码发布于 https://github.com/HVision-NKU/DenseVLM。