When we look around and perform complex tasks, how we see and selectively process what we see is crucial. However, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce $\textit{V}^*$, an LLM-guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise targeting of specific visual elements. This integration results in a new MLLM meta-architecture, named $\textbf{S}$how, s$\textbf{EA}$rch, and Tel$\textbf{L}$ (SEAL). We further create $\textit{V}^*$Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available https://github.com/penghao-wu/vstar.

翻译：当我们环顾四周并执行复杂任务时，如何观察以及如何有选择性地加工所见信息至关重要。然而，当前多模态大语言模型（MLLMs）缺乏这种视觉搜索机制，这严重削弱了它们聚焦重要视觉细节的能力，尤其是在处理高分辨率且视觉拥挤的图像时。为解决这一问题，我们提出$\textit{V}^*$——一种由大语言模型引导的视觉搜索机制，该机制利用LLM中的世界知识实现高效视觉查询。当与MLLM结合时，该机制可增强协同推理、上下文理解能力，并实现对特定视觉元素的精准定位。这一整合催生了名为$\textbf{S}$how（展示）、s$\textbf{EA}$rch（搜索）和Tel$\textbf{L}$（叙述）的新MLLM元架构（SEAL）。我们还构建了$\textit{V}^*$Bench基准测试集，专门评估MLLM处理高分辨率图像和聚焦视觉细节的能力。本研究凸显了将视觉搜索能力融入多模态系统的必要性。代码开源地址：https://github.com/penghao-wu/vstar。