Conventional object detectors rely on cross-entropy classification, which can be vulnerable to class imbalance and label noise. We propose CLIP-Joint-Detect, a simple and detector-agnostic framework that integrates CLIP-style contrastive vision-language supervision through end-to-end joint training. A lightweight parallel head projects region or grid features into the CLIP embedding space and aligns them with learnable class-specific text embeddings via InfoNCE contrastive loss and an auxiliary cross-entropy term, while all standard detection losses are optimized simultaneously. The approach applies seamlessly to both two-stage and one-stage architectures. We validate it on Pascal VOC 2007+2012 using Faster R-CNN and on the large-scale MS COCO 2017 benchmark using modern YOLO detectors (YOLOv11), achieving consistent and substantial improvements while preserving real-time inference speed. Extensive experiments and ablations demonstrate that joint optimization with learnable text embeddings markedly enhances closed-set detection performance across diverse architectures and datasets.

翻译：传统的目标检测器依赖于交叉熵分类，这可能容易受到类别不平衡和标签噪声的影响。我们提出了CLIP-Joint-Detect，这是一个简单且与检测器无关的框架，它通过端到端联合训练集成了CLIP风格的对比视觉-语言监督。一个轻量级的并行头将区域或网格特征投影到CLIP嵌入空间中，并通过InfoNCE对比损失和一个辅助交叉熵项，将它们与可学习的类别特定文本嵌入对齐，同时所有标准检测损失被同步优化。该方法可无缝应用于两阶段和一阶段架构。我们在Pascal VOC 2007+2012数据集上使用Faster R-CNN，以及在大规模MS COCO 2017基准上使用现代YOLO检测器（YOLOv11）进行了验证，在保持实时推理速度的同时，取得了持续且显著的性能提升。大量的实验和消融研究表明，使用可学习文本嵌入进行联合优化，显著提升了不同架构和数据集上的闭集检测性能。