Despite impressive advances in recent multimodal large language models (MLLMs), state-of-the-art models such as from the GPT-4 suite still struggle with knowledge-intensive tasks. To address this, we consider Reverse Image Retrieval (RIR) augmented generation, a simple yet effective strategy to augment MLLMs with web-scale reverse image search results. RIR robustly improves knowledge-intensive visual question answering (VQA) of GPT-4V by 37-43%, GPT-4 Turbo by 25-27%, and GPT-4o by 18-20% in terms of open-ended VQA evaluation metrics. To our surprise, we discover that RIR helps the model to better access its own world knowledge. Concretely, our experiments suggest that RIR augmentation helps by providing further visual and textual cues without necessarily containing the direct answer to a query. In addition, we elucidate cases in which RIR can hurt performance and conduct a human evaluation. Finally, we find that the overall advantage of using RIR makes it difficult for an agent that can choose to use RIR to perform better than an approach where RIR is the default setting.

翻译：尽管近年来多模态大语言模型（MLLMs）取得了令人瞩目的进展，但以GPT-4系列为代表的最先进模型在处理知识密集型任务时仍面临困难。为解决这一问题，我们提出反向图像检索（RIR）增强生成策略，这是一种通过网页级反向图像搜索结果增强MLLM能力的简单而有效的方法。在开放式视觉问答（VQA）评估指标下，RIR将GPT-4V的知识密集型VQA性能稳健提升了37-43%，GPT-4 Turbo提升了25-27%，GPT-4o提升了18-20%。令人惊讶的是，我们发现RIR能帮助模型更好地访问其自身的世界知识。具体而言，实验表明RIR增强通过提供额外的视觉和文本线索发挥作用，而这些线索并不一定包含查询的直接答案。此外，我们阐明了RIR可能损害性能的情况并进行了人工评估。最后，我们发现使用RIR带来的整体优势使得能够自主选择是否使用RIR的智能体，难以超越将RIR作为默认设置的方案。