Multimodal reasoning stands as a pivotal capability for large vision-language models (LVLMs). The integration with Domain-Specific Languages (DSL), offering precise visual representations, equips these models with the opportunity to execute more accurate reasoning in complex and professional domains. However, the vanilla Chain-of-Thought (CoT) prompting method faces challenges in effectively leveraging the unique strengths of visual and DSL representations, primarily due to their differing reasoning mechanisms. Additionally, it often falls short in addressing critical steps in multi-step reasoning tasks. To mitigate these challenges, we introduce the \underline{B}i-Modal \underline{B}ehavioral \underline{A}lignment (BBA) prompting method, designed to maximize the potential of DSL in augmenting complex multi-modal reasoning tasks. This method initiates by guiding LVLMs to create separate reasoning chains for visual and DSL representations. Subsequently, it aligns these chains by addressing any inconsistencies, thus achieving a cohesive integration of behaviors from different modalities. Our experiments demonstrate that BBA substantially improves the performance of GPT-4V(ision) on geometry problem solving ($28.34\% \to 34.22\%$), chess positional advantage prediction ($42.08\% \to 46.99\%$) and molecular property prediction ($77.47\% \to 83.52\%$).

翻译：多模态推理是大型视觉-语言模型（LVLMs）的关键能力。通过与提供精确视觉表示的领域特定语言（DSL）相融合，这类模型得以在复杂专业领域执行更精准的推理任务。然而，传统思维链（CoT）提示方法因视觉与DSL表征的推理机制存在本质差异，难以有效发挥二者的独特优势，且在多步推理任务中常无法应对关键步骤的挑战。为缓解这些问题，我们提出双模态行为对齐（BBA）提示方法，旨在最大化DSL在增强复杂多模态推理任务中的潜力。该方法首先引导LVLMs分别构建基于视觉表征和DSL表征的独立推理链，继而通过消除链间不一致性实现对齐，最终达成不同模态行为特征的有机融合。实验表明，BBA方法显著提升了GPT-4V(ision)在几何问题求解（28.34%→34.22%）、棋局优势预测（42.08%→46.99%）和分子属性预测（77.47%→83.52%）任务中的性能表现。