We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.

翻译：我们提出Lumos，首个具备文本理解能力的端到端多模态问答系统。其核心组件为场景文本识别（STR）模块，该模块能从第一人称视角图像中提取文本信息，其输出结果被用于增强多模态大语言模型（MM-LLM）的输入。在构建Lumos过程中，我们面临了与STR质量、系统延迟及模型推理相关的诸多挑战。本文深入探讨这些挑战，并阐述为克服这些困难所采用的系统架构、设计选型及建模技术。我们同时对各组件进行了全面评估，展示了系统的高质量与高效率。