Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.

翻译：视觉-语言（VL）模型已成为研究焦点，推动了多模态推理领域的显著进展。此类架构通常包含视觉编码器、大型语言模型（LLM）以及将视觉特征与LLM表示空间对齐的投影模块。尽管取得显著成功，但存在关键局限：视觉编码过程与用户查询（通常以图像相关问题形式呈现）相互解耦。这导致生成的视觉特征难以最优适配查询关注的图像元素。为此，我们提出QA-ViT——一种面向问题的视觉Transformer方法，通过将问题感知机制直接嵌入视觉编码器，实现动态视觉特征聚焦于与问题相关的图像区域。QA-ViT具有模型无关性，可高效集成至任意VL架构。大量实验证明，该方法能有效适配多种多模态架构，在不同任务中持续提升性能，展现出增强视觉与场景文字理解的潜力。