Training a Large Visual Language Model (LVLM) from scratch, like GPT-4, is resource-intensive. Our paper presents a play-and-plug module for Large Language Models (LLMs), namely Interactive Perception Network (IPN), aiming to achieve a LVLM by incorporating the image understanding capability into LLMs. Previous methods incorporate visual information into LLMs with a simple visual mapping network, where the image feature is projected into the embedding space of LLMs via a linear layer. Such mapping network projects the image feature once yet does not consider the interaction between the image and the human input query. Hence, the obtained visual information with no connections with human intention may be inadequate for LLMs to make intention-following responses, which we term as static visual information. IPN addresses this issue by allowing the LLM to request the desired visual information aligned with various human instructions, which we term as the dynamic interaction between the LLM and visual information. Specifically, IPN consists of a simple visual mapping network to provide the basic perception of an image for LLMs. It also contains additional modules responsible for acquiring requests from LLMs, performing request-based visual information interaction, and transmitting the resulting interacted visual information to LLMs, respectively. In this way, LLMs act to understand the human query, deliver the corresponding request to the request-based visual information interaction module, and generate the response based on the interleaved multimodal information. We evaluate IPN through extensive experiments on multimodal question answering, reasoning, and so on, demonstrating that it significantly improves the zero-shot performance of LVLMs on various multimodal tasks compared to previous methods.

翻译：从头训练大型视觉语言模型（如 GPT-4）资源消耗巨大。本文提出一种即插即用的大语言模型（LLM）模块，即交互式感知网络（IPN），旨在通过将图像理解能力融入 LLM，构建大型视觉语言模型（LVLM）。现有方法通过简单视觉映射网络将视觉信息引入 LLM，即利用线性层将图像特征投影至 LLM 的嵌入空间。此类映射网络对图像特征仅做一次性投影处理，未考虑图像与人类输入查询之间的交互。因此，所获取的视觉信息与人类意图缺乏关联，可能不足以让 LLM 生成符合意图的响应——我们将其定义为静态视觉信息。IPN 通过允许 LLM 请求与不同人类指令对齐的所需视觉信息来解决该问题，我们称之为 LLM 与视觉信息间的动态交互。具体而言，IPN 包含一个为 LLM 提供图像基础感知的简单视觉映射网络，同时还配备三个附加模块，分别负责获取 LLM 的请求、执行基于请求的视觉信息交互，以及将交互后的视觉信息传递给 LLM。由此，LLM 得以理解人类查询，向基于请求的视觉信息交互模块发送相应指令，并基于交织的多模态信息生成响应。我们通过多模态问答、推理等任务的广泛实验评估 IPN，结果表明，与现有方法相比，IPN 显著提升了 LVLM 在各种多模态任务上的零样本性能。