Vision-Language Models (VLMs) have achieved remarkable progress in complex visual understanding across scientific and reasoning tasks. While performance benchmarking has advanced our understanding of these capabilities, the critical dimension of uncertainty quantification has received insufficient attention. Therefore, unlike prior conformal prediction studies that focused on limited settings, we conduct a comprehensive uncertainty benchmarking study, evaluating 18 state-of-the-art VLMs (open and closed-source) across 6 multimodal datasets with 3 distinct scoring functions. For closed-source models lacking token-level logprob access, we develop and validate instruction-guided likelihood proxies. Our findings demonstrate that larger models consistently exhibit better uncertainty quantification; models that know more also know better what they don't know. More certain models achieve higher accuracy, while mathematical and reasoning tasks elicit poorer uncertainty performance across all models compared to other domains. This work establishes a foundation for reliable uncertainty evaluation in multimodal systems.

翻译：视觉语言模型（VLMs）在科学和推理任务中的复杂视觉理解方面取得了显著进展。尽管性能基准测试增进了我们对这些能力的理解，但不确定性量化这一关键维度尚未得到充分关注。因此，不同于先前局限于有限场景的共形预测研究，我们开展了一项全面的不确定性基准测试研究，使用3种不同的评分函数，在6个多模态数据集上评估了18个最先进的VLM（包括开源和闭源模型）。针对缺乏词元级对数概率访问权限的闭源模型，我们开发并验证了指令引导的似然代理方法。我们的研究结果表明：更大规模的模型始终表现出更好的不确定性量化能力；知道更多的模型也更清楚自己不知道什么。不确定性更低的模型能获得更高的准确率；而与其他领域相比，所有模型在数学和推理任务上的不确定性表现均较差。这项工作为多模态系统中可靠的不确定性评估奠定了基础。