The visual commonsense reasoning (VCR) task is to choose an answer and provide a justifying rationale based on the given image and textural question. Representative works first recognize objects in images and then associate them with key words in texts. However, existing approaches do not consider exact positions of objects in a human-like three-dimensional (3D) manner, making them incompetent to accurately distinguish objects and understand visual relation. Recently, multi-modal large language models (MLLMs) have been used as powerful tools for several multi-modal tasks but not for VCR yet, which requires elaborate reasoning on specific visual objects referred by texts. In light of the above, an MLLM enhanced pseudo 3D perception framework is designed for VCR. Specifically, we first demonstrate that the relation between objects is relevant to object depths in images, and hence introduce object depth into VCR frameworks to infer 3D positions of objects in images. Then, a depth-aware Transformer is proposed to encode depth differences between objects into the attention mechanism of Transformer to discriminatively associate objects with visual scenes guided by depth. To further associate the answer with the depth of visual scene, each word in the answer is tagged with a pseudo depth to realize depth-aware association between answer words and objects. On the other hand, BLIP-2 as an MLLM is employed to process images and texts, and the referring expressions in texts involving specific visual objects are modified with linguistic object labels to serve as comprehensible MLLM inputs. Finally, a parameter optimization technique is devised to fully consider the quality of data batches based on multi-level reasoning confidence. Experiments on the VCR dataset demonstrate the superiority of the proposed framework over state-of-the-art approaches.

翻译：视觉常识推理（VCR）任务旨在根据给定的图像和文本问题选择答案并提供合理的解释。代表性方法首先识别图像中的物体，然后将其与文本中的关键词关联。然而，现有方法未以类人三维（3D）方式考虑物体的精确位置，导致其无法准确区分物体并理解视觉关系。近期，多模态大语言模型（MLLMs）已被用于多项多模态任务，但尚未应用于VCR，后者需要对文本所指的特定视觉物体进行精细推理。鉴于上述问题，本文设计了一种MLLM增强的伪三维感知框架用于VCR。具体而言，我们首先证明物体间关系与图像中物体的深度相关，从而将物体深度引入VCR框架以推断图像中物体的3D位置。随后，提出了一种深度感知Transformer，将物体间的深度差异编码到Transformer的注意力机制中，以在深度引导下区分性地将物体与视觉场景关联。为进一步将答案与视觉场景深度关联，答案中的每个词被赋予伪深度标签，以实现答案词与物体间的深度感知关联。另一方面，采用BLIP-2作为MLLM处理图像与文本，并将文本中涉及特定视觉物体的指代表达替换为可理解的语言化物体标签，作为MLLM的输入。最后，设计了一种参数优化技术，基于多级推理置信度全面考虑数据批次质量。在VCR数据集上的实验表明，所提出的框架优于现有最先进方法。