Referring Expressions Generation (REG) aims to produce textual descriptions that unambiguously identifies specific objects within a visual scene. Traditionally, this has been achieved through supervised learning methods, which perform well on specific data distributions but often struggle to generalize to new images and concepts. To address this issue, we present a novel approach for REG, named DisCLIP, short for discriminative CLIP. We build on CLIP, a large-scale visual-semantic model, to guide an LLM to generate a contextual description of a target concept in an image while avoiding other distracting concepts. Notably, this optimization happens at inference time and does not require additional training or tuning of learned parameters. We measure the quality of the generated text by evaluating the capability of a receiver model to accurately identify the described object within the scene. To achieve this, we use a frozen zero-shot comprehension module as a critique of our generated referring expressions. We evaluate DisCLIP on multiple referring expression benchmarks through human evaluation and show that it significantly outperforms previous methods on out-of-domain datasets. Our results highlight the potential of using pre-trained visual-semantic models for generating high-quality contextual descriptions.

翻译：论文摘要：指代表达生成（REG）旨在生成能够明确识别视觉场景中特定对象的文本描述。传统上，这通过监督学习方法实现，这些方法在特定数据分布上表现良好，但往往难以泛化到新图像和概念。为解决这一问题，我们提出了一种新颖的REG方法，名为DisCLIP（判别式CLIP）。我们基于大规模视觉-语义模型CLIP，引导大语言模型（LLM）生成图像中目标概念的上下文描述，同时避免其他干扰概念。值得注意的是，这一优化过程在推理时完成，无需额外训练或调整学习参数。我们通过评估接收模型能否准确识别场景中被描述对象来衡量生成文本的质量。为此，我们使用冻结的零样本理解模块作为所生成指代表达的评判标准。通过人工评估，我们在多个指代表达基准上对DisCLIP进行评测，结果表明其在跨领域数据集上显著优于以往方法。我们的研究结果凸显了利用预训练视觉-语义模型生成高质量上下文描述的潜力。