The remarkable advancements in Multimodal Large Language Models (MLLMs) have not rendered them immune to challenges, particularly in the context of handling deceptive information in prompts, thus producing hallucinated responses under such conditions. To quantitatively assess this vulnerability, we present MAD-Bench, a carefully curated benchmark that contains 850 test samples divided into 6 categories, such as non-existent objects, count of objects, spatial relationship, and visual confusion. We provide a comprehensive analysis of popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps between GPT-4V and other models; and previous robust instruction-tuned models, such as LRV-Instruction and LLaVA-RLHF, are not effective on this new benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of any other model in our experiments ranges from 5% to 35%. We further propose a remedy that adds an additional paragraph to the deceptive prompts to encourage models to think twice before answering the question. Surprisingly, this simple method can even double the accuracy; however, the absolute numbers are still too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark to stimulate further research to enhance models' resilience against deceptive prompts.

翻译：多模态大语言模型（MLLMs）的显著进步并未使其免受挑战，尤其在处理提示中的欺骗性信息时，模型会产生幻觉式响应。为定量评估这一脆弱性，我们提出了MAD-Bench——一个精心策划的基准测试集，包含850个测试样本，分为6个类别（如不存在的对象、对象计数、空间关系、视觉混淆等）。我们对主流MLLMs进行了全面分析，涵盖GPT-4V、Gemini-Pro到开源模型（如LLaVA-1.5和CogVLM）。实验发现，GPT-4V与其他模型之间存在显著性能差距；而此前具有鲁棒性的指令微调模型（如LRV-Instruction和LLaVA-RLHF）在此新基准上效果不佳。GPT-4V在MAD-Bench上达到75.02%的准确率，而其他模型的准确率仅介于5%至35%之间。我们进一步提出一种补救措施：在欺骗性提示中添加额外段落，引导模型在回答问题前进行二次思考。令人惊讶的是，这种简单方法可使准确率翻倍，但绝对数值仍不足以令人满意。我们期望MAD-Bench能成为有价值的基准，推动后续研究以增强模型对欺骗性提示的鲁棒性。