Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. For researchers interested in further exploration, our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.

翻译：视觉语言模型（VLM）通过融合视觉理解能力扩展了大语言模型（LLM），在开放式视觉问答（VQA）任务中展现出显著进步。然而，这些模型难以准确解读现实场景中普遍存在的含文本图像。标准图像信息提取流程通常涉及学习一组固定的查询嵌入，这些嵌入旨在封装图像上下文，随后作为软提示输入至LLM。但该过程受限于标记数量，可能削弱对富含文本场景的识别能力。为改进此局限，本研究提出BLIVA：一种结合视觉助手的增强版InstructBLIP。BLIVA融合了InstructBLIP的查询嵌入，并借鉴LLaVA技术，将编码后的图像块嵌入直接投影至LLM。该方法有助于模型捕获查询解码过程中可能遗漏的复杂细节。实验证据表明，与基线模型InstructBLIP相比，我们的模型BLIVA在处理富含文本的VQA基准测试（OCR-VQA基准上提升高达17.76%）及通用（非特定文本丰富）VQA基准测试（视觉空间推理基准上提升高达7.9%）中表现显著增强。无论图像是否含文本，BLIVA均展现出对现实世界图像强大的解码能力。为展示BLIVA的广泛工业应用潜力，我们采用涵盖11个多样化类别的YouTube缩略图及其问答对构成的新数据集对模型进行评估。为便于研究者进一步探索，我们的代码与模型已在https://github.com/mlpc-ucsd/BLIVA 开源。