Vision-language models (VLMs) can produce confident visual answers even when the required visual evidence is missing, blank, or unrelated to the question. This failure mode, recently described as mirage (mirage2026), is especially concerning in medical and document VQA, where a plausible but visually ungrounded answer may be mistaken for image-based evidence. We study the complementary problem of pre-release mirage detection: given an image-question pair, determine whether the VLM should answer or abstain before generation. To that end, we propose a novel model-agnostic Text-Conditioned Layer-wise Internal Alignment (TC-LIA) method that probes patch-token representations across the layers of a CLIP ViT-H/14 vision encoder. The key idea is to project layer-wise image patch tokens into the final CLIP embedding space and measure their similarity with the question embedding, thereby tracking whether question-relevant visual evidence emerges across vision layers. TC-LIA summarizes this alignment trajectory using final image-text cosine similarity, late-layer top-k patch-text alignment, early-to-late gain, and layer-wise slope. These features are combined with pixel-statistic based blank/noise detection, zero-shot domain routing, and structured VLM self-assessment in an ensemble. Across five VQA domains with related, unrelated-real, and blank/noise inputs, and across twelve VLM backbones, Qwen2.5-VL-32B achieves the highest three-class detection accuracy of 94.7% with a 3.0% mirage rate, while Qwen2.5-VL-72B achieves 94.6% accuracy with a lower 2.8% mirage rate. Baseline mirage rates span 21.7-66.6%.
翻译:视觉语言模型(VLM)即使在缺少、空白或与问题无关的视觉证据时,也能生成看似自信的视觉答案。这种最近被描述为“幻象”(mirage2026)的失败模式,在医学和文档视觉问答中尤为令人担忧,因为一个看似合理但缺乏视觉依据的答案可能被误认为基于图像的证据。我们研究了预生成式幻象检测这一互补性问题:给定一个图像-问题对,在模型生成之前判断VLM是否应回答或弃权。为此,我们提出了一种新颖的、与模型无关的文本条件层间内部对齐(TC-LIA)方法,该方法探测CLIP ViT-H/14视觉编码器各层中的图像块-标记表征。其核心思想是将逐层的图像块标记投影到最终的CLIP嵌入空间,并测量它们与问题嵌入的相似度,从而追踪与问题相关的视觉证据是否在各视觉层中涌现。TC-LIA通过最终图像-文本余弦相似度、高层top-k图像块-文本对齐、从低层到高层的增益以及逐层斜率来总结这一对齐轨迹。这些特征与基于像素统计的空白/噪声检测、零样本领域路由以及结构化VLM自我评估相结合,形成集成方法。在五个包含相关、无关真实及空白/噪声输入的VQA领域,以及十二个VLM主干网络上,Qwen2.5-VL-32B实现了最高的三分类检测准确率94.7%,幻象率3.0%,而Qwen2.5-VL-72B实现了94.6%的准确率,幻象率更低,为2.8%。基线幻象率范围为21.7%-66.6%。