When people query Vision-Language Models (VLMs) but cannot see the accompanying visual context (e.g. for blind and low-vision users), augmenting VLM predictions with natural language explanations can signal which model predictions are reliable. However, prior work has found that explanations can easily convince users that inaccurate VLM predictions are correct. To remedy undesirable overreliance on VLM predictions, we propose evaluating two complementary qualities of VLM-generated explanations via two quality scoring functions. We propose Visual Fidelity, which captures how faithful an explanation is to the visual context, and Contrastiveness, which captures how well the explanation identifies visual details that distinguish the model's prediction from plausible alternatives. On the A-OKVQA, VizWiz, and MMMU-Pro tasks, these quality scoring functions are better calibrated with model correctness than existing explanation qualities. We conduct a user study in which participants have to decide whether a VLM prediction is accurate without viewing its visual context. We observe that showing our quality scores alongside VLM explanations improves participants' accuracy at predicting VLM correctness by 11.1%, including a 15.4% reduction in the rate of falsely believing incorrect predictions. These findings highlight the utility of explanation quality scores in fostering appropriate reliance on VLM predictions.

翻译：当用户查询视觉-语言模型（VLMs）却无法获取伴随的视觉上下文（例如视障与低视力用户）时，为VLM预测补充自然语言解释能够指示哪些模型预测是可靠的。然而，先前研究发现，解释可能轻易使用户确信不准确的VLM预测是正确的。为纠正对VLM预测的不良过度依赖，我们提出通过两种质量评分函数来评估VLM生成解释的两个互补特性：视觉保真度——衡量解释对视觉上下文的忠实程度；对比性——衡量解释识别视觉细节以区分模型预测与合理替代方案的能力。在A-OKVQA、VizWiz和MMMU-Pro任务上的实验表明，相较于现有解释质量指标，这些质量评分函数与模型正确率的校准性更优。我们开展了一项用户研究，要求参与者在未观察视觉上下文的情况下判断VLM预测的准确性。结果显示，在VLM解释旁展示我们的质量评分可使参与者预测VLM正确率的准确度提升11.1%，其中错误相信不准确预测的比例降低15.4%。这些发现凸显了解释质量评分在促进对VLM预测形成合理依赖方面的实用价值。