LLaVA-Interactive is a research prototype for multimodal human-AI interaction. The system can have multi-turn dialogues with human users by taking multimodal user inputs and generating multimodal responses. Importantly, LLaVA-Interactive goes beyond language prompt, where visual prompt is enabled to align human intents in the interaction. The development of LLaVA-Interactive is extremely cost-efficient as the system combines three multimodal skills of pre-built AI models without additional model training: visual chat of LLaVA, image segmentation from SEEM, as well as image generation and editing from GLIGEN. A diverse set of application scenarios is presented to demonstrate the promises of LLaVA-Interactive and to inspire future research in multimodal interactive systems.

翻译：LLaVA-Interactive是一个面向多模态人机交互的研究原型系统。该支持通过接收用户的多模态输入并生成多模态响应，与人类用户进行多轮对话。值得注意的是，LLaVA-Interactive超越了传统语言提示的局限，通过引入视觉提示机制实现人类意图的精准对齐。该系统在开发过程中展现出极高的成本效益——无需额外模型训练，仅需整合三个预构建AI模型的多模态技能：LLaVA的视觉对话能力、SEEM的图像分割功能，以及GLIGEN的图像生成与编辑能力。本文展示了多样化的应用场景，验证了LLaVA-Interactive的潜力，并激励未来多模态交互系统的研究。