Multimodal Large Language Models (LLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether multimodal LLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to $46\%$ with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose three automatic visual cropping methods as inference time mechanisms to improve the zero-shot performance of multimodal LLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that multimodal LLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance. Our code and data are publicly available.

翻译：多模态大语言模型（LLMs）近期在视觉问答（VQA）任务上取得了令人瞩目的零样本准确率——该基础任务影响着众多下游应用与领域。鉴于这些模型的广泛应用潜力，探究其在处理不同图像与问题属性时的局限性至关重要。本研究探讨多模态LLMs能否同等感知图像中的微小细节与宏观细节。具体而言，我们发现模型回答视觉问题的零样本准确率对问题中视觉主体的大小极为敏感，准确率随主体尺寸减小而下降高达46%。通过观察人类视觉裁剪能够显著缓解模型对尺寸的敏感性，我们进一步证实了这种效应的因果性。受人类裁剪有效性的启发，我们提出三种自动视觉裁剪方法作为推理阶段机制，以提升多模态LLMs的零样本性能。我们在四个主流VQA数据集及VQAv2数据集中针对精细视觉细节的子集上验证了其有效性。研究结果表明，在细节敏感的VQA应用中需谨慎使用多模态LLMs，而视觉裁剪是提升其零样本性能的可行方向。我们的代码与数据已公开。