Retrieval augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for combined understanding of multimodal data such as text, images and videos is appealing but two critical limitations exist: one-time, upfront capture of all content in large multimodal data as text descriptions entails high processing times, and not all information in the rich multimodal data is typically in the text descriptions. Since the user queries are not known apriori, developing a system for multimodal to text conversion and interactive querying of multimodal data is challenging. To address these limitations, we propose iRAG, which augments RAG with a novel incremental workflow to enable interactive querying of large corpus of multimodal data. Unlike traditional RAG, iRAG quickly indexes large repositories of multimodal data, and in the incremental workflow, it uses the index to opportunistically extract more details from select portions of the multimodal data to retrieve context relevant to an interactive user query. Such an incremental workflow avoids long multimodal to text conversion times, overcomes information loss issues by doing on-demand query-specific extraction of details in multimodal data, and ensures high quality of responses to interactive user queries that are often not known apriori. To the best of our knowledge, iRAG is the first system to augment RAG with an incremental workflow to support efficient interactive querying of large, real-world multimodal data. Experimental results on real-world long videos demonstrate 23x to 25x faster video to text ingestion, while ensuring that quality of responses to interactive user queries is comparable to responses from a traditional RAG where all video data is converted to text upfront before any querying.

翻译：检索增强生成（RAG）系统融合了语言生成与信息检索的优势，为聊天机器人等众多实际应用提供支撑。将RAG用于文本、图像和视频等多模态数据的联合理解具有吸引力，但存在两个关键局限：一是对大容量多模态数据中全部内容进行一次性前置捕获为文本描述会导致处理时间过长，二是多模态数据中的丰富信息往往无法完整体现在文本描述中。由于用户查询具有先验未知性，开发多模态到文本转换及交互式查询的系统面临挑战。为此，我们提出iRAG，该方案通过为RAG新增增量式工作流，实现对大规模多模态语料库的交互式查询。与传统RAG不同，iRAG快速索引多模态数据的大容量存储库，在增量式工作流中利用该索引从多模态数据选定部分时机性地提取更多细节，以检索与交互式用户查询相关的上下文。这种增量式工作流避免了多模态到文本的长时间转换，通过按需针对查询提取多模态数据细节克服信息丢失问题，并确保对常具先验未知性的交互式用户查询提供高质量响应。据我们所知，iRAG是首个为RAG引入增量式工作流以支持高效交互式查询大规模实际多模态数据的系统。在真实长视频上的实验结果表明，视频到文本的注入速度提升23至25倍，同时确保对交互式用户查询的响应质量与所有视频数据在查询前预先转换为文本的传统RAG方案相当。