Despite making significant progress in multi-modal tasks, current Multi-modal Large Language Models (MLLMs) encounter the significant challenge of hallucinations, which may lead to harmful consequences. Therefore, evaluating MLLMs' hallucinations is becoming increasingly important in model improvement and practical application deployment. Previous works are limited in high evaluation costs (e.g., relying on humans or advanced LLMs) and insufficient evaluation dimensions (e.g., types of tasks and hallucinations). In this paper, we propose an LLM-free multi-dimensional benchmark AMBER, which can be used to evaluate both generative task and discriminative task including existence, attribute and relation hallucination. Based on AMBER, we design a low-cost and efficient evaluation pipeline. Additionally, we conduct a comprehensive evaluation and detailed analysis of mainstream MLLMs including GPT-4V(ision), and also give guideline suggestions for mitigating hallucinations. The data and code of AMBER are available at https://github.com/junyangwang0410/AMBER.

翻译：尽管多模态任务取得了显著进展，当前多模态大语言模型仍面临幻觉这一重大挑战，可能导致有害后果。因此，评估多模态大语言模型的幻觉问题在模型改进和实际应用部署中日益重要。以往研究存在评估成本高（如依赖人工或高级大语言模型）和评估维度不足（如任务类型和幻觉种类）的局限性。本文提出了一种免大语言模型的多维度基准AMBER，可同时评估生成式任务和判别式任务，涵盖存在性幻觉、属性幻觉和关系幻觉。基于AMBER，我们设计了一套低成本、高效的评估流程。此外，我们对包括GPT-4V(ision)在内的主流多模态大语言模型进行了全面评估与详细分析，并给出了缓解幻觉的指导性建议。AMBER的数据和代码已在https://github.com/junyangwang0410/AMBER开源。