Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabilities. Firstly, Monkey processes input images by dividing them into uniform patches, each matching the size (e.g., 448x448) used in the original training of the well-trained vision encoder. Equipped with individual adapter for each patch, Monkey can handle higher resolutions up to 1344x896 pixels, enabling the detailed capture of complex visual information. Secondly, it employs a multi-level description generation method, enriching the context for scene-object associations. This two-part strategy ensures more effective learning from generated data: the higher resolution allows for a more detailed capture of visuals, which in turn enhances the effectiveness of comprehensive descriptions. Extensive ablative results validate the effectiveness of our designs. Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats. Specially, in qualitative tests focused on dense text question answering, Monkey has exhibited encouraging results compared with GPT4V. Code is available at https://github.com/Yuliang-Liu/Monkey.

翻译：大型多模态模型（LMMs）在视觉-语言任务中展现出潜力，但在高分辨率输入和细粒度场景理解方面仍面临挑战。为应对这些问题，我们提出Monkey模型以增强LMM能力。首先，Monkey通过将输入图像分割为统一块（patch），每个块的大小（例如448×448）与预训练视觉编码器的原始训练尺寸匹配。通过为每个块配备独立适配器，Monkey可处理高达1344×896像素的分辨率，从而精细捕捉复杂视觉信息。其次，采用多层级描述生成方法，丰富场景-对象关联的上下文。这种双阶段策略能更有效地利用生成数据进行学习：更高分辨率支持更精细的视觉捕捉，进而增强综合描述的有效性。大量消融实验验证了设计的有效性。此外，在18个数据集上的实验表明，Monkey在图像描述和多种视觉问答格式等多项任务中优于现有LMM。特别在聚焦密集文本问答的定性测试中，Monkey展现了与GPT4V相比令人鼓舞的结果。代码已开源：https://github.com/Yuliang-Liu/Monkey。