Vision-language models (VLMs) are increasingly used as automated judges for multimodal systems, yet their scores provide no indication of reliability. We study this problem through conformal prediction, a distribution-free framework that converts a judge's point score into a calibrated prediction interval using only score-token log-probabilities, with no retraining. We present the first systematic analysis of conformal prediction for VLM-as-a-Judge across 3 judges and 14 visual task categories. Our results show that evaluation uncertainty is strongly task-dependent: intervals cover ~40% of the score range for aesthetics and natural images but expand to ~70% for chart and mathematical reasoning, yielding a quantitative reliability map for multimodal evaluation. We further identify a failure mode not captured by standard evaluation metrics, ranking-scoring decoupling, where judges achieve high ranking correlation while producing wide, uninformative intervals, correctly ordering responses but failing to assign reliable absolute scores. Finally, we show that interval width is driven primarily by task difficulty and annotation quality, i.e., the same judge and method yield 4.5x narrower intervals on a clean, multi-annotator captioning benchmark. Code: https://github.com/divake/VLM-Judge-Uncertainty

翻译：[translated abstract in Chinese] 视觉语言模型（VLM）正越来越多地被用作多模态系统的自动评估裁判，但其评分并未提供可靠性的任何指示。我们通过共形预测来研究这一问题，这是一种无分布假设的框架，仅利用评分令牌的对数概率即可将裁判的点估计值转换为校准后的预测区间，无需重新训练。我们首次系统分析了共形预测在VLM裁判中的表现，涵盖3个裁判模型和14类视觉任务。结果显示，评估不确定性具有强烈的任务依赖性：对于美学和自然图像，区间覆盖评分范围的约40%；而对于图表和数学推理任务，区间则扩大至约70%，从而形成了多模态评估的定量可靠性图谱。我们进一步识别出标准评估指标无法捕获的失效模式——排序-评分解耦：裁判在实现高排序相关性的同时，产生了宽泛无信息量的区间，即能正确排序响应却无法分配可靠的绝对评分。最后，我们证明区间宽度主要由任务难度和标注质量驱动：同一裁判和方法在干净且含有多标注者的图像描述基准上，区间宽度可缩小4.5倍。代码：https://github.com/divake/VLM-Judge-Uncertainty