Hallucination detection in captions (HalDec) assesses a vision-language model's ability to correctly align image content with text by identifying errors in captions that misrepresent the image. Beyond evaluation, effective hallucination detection is also essential for curating high-quality image-caption pairs used to train VLMs. However, the generalizability of VLMs as hallucination detectors across different captioning models and hallucination types remains unclear due to the lack of a comprehensive benchmark. In this work, we introduce HalDec-Bench, a benchmark designed to evaluate hallucination detectors in a principled and interpretable manner. HalDec-Bench contains captions generated by diverse VLMs together with human annotations indicating the presence of hallucinations, detailed hallucination-type categories, and segment-level labels. The benchmark provides tasks with a wide range of difficulty levels and reveals performance differences across models that are not visible in existing multimodal reasoning or alignment benchmarks. Our analysis further uncovers two key findings. First, detectors tend to recognize sentences appearing at the beginning of a response as correct, regardless of their actual correctness. Second, our experiments suggest that dataset noise can be substantially reduced by using strong VLMs as filters while employing recent VLMs as caption generators. Our project page is available at https://dahlian00.github.io/HalDec-Bench-Page/.

翻译：摘要：描述中的幻觉检测（HalDec）通过识别描述中扭曲图像内容的错误，评估视觉-语言模型正确对齐图像内容与文本的能力。除评估外，有效的幻觉检测对于筛选用于训练VLM的高质量图像-描述对也至关重要。然而，由于缺乏全面的基准测试，VLM作为幻觉检测器在不同描述模型和幻觉类型间的泛化能力尚不明确。本研究引入HalDec-Bench，一个旨在以原则性且可解释的方式评估幻觉检测器的基准测试。HalDec-Bench包含由多样化VLM生成的描述，以及人类标注的幻觉存在性、详细幻觉类型类别和片段级标签。该基准测试提供多难度任务，并揭示了现有多模态推理或对齐基准测试中未显现的模型间性能差异。我们的分析进一步揭示两项关键发现：其一，检测器倾向于将出现在响应开头的句子判定为正确，不论其实际正确性；其二，实验表明，使用强VLM作为过滤器、同时采用近期VLM作为描述生成器，可显著降低数据集噪声。项目主页见https://dahlian00.github.io/HalDec-Bench-Page/。