Reasoning ability is one of the most crucial capabilities of a foundation model, signifying its capacity to address complex reasoning tasks. Chain-of-Thought (CoT) technique is widely regarded as one of the effective methods for enhancing the reasoning ability of foundation models and has garnered significant attention. However, the reasoning process of CoT is linear, step-by-step, similar to personal logical reasoning, suitable for solving general and slightly complicated problems. On the contrary, the thinking pattern of an expert owns two prominent characteristics that cannot be handled appropriately in CoT, i.e., high-order multi-hop reasoning and multimodal comparative judgement. Therefore, the core motivation of this paper is transcending CoT to construct a reasoning paradigm that can think like an expert. The hyperedge of a hypergraph could connect various vertices, making it naturally suitable for modelling high-order relationships. Inspired by this, this paper innovatively proposes a multimodal Hypergraph-of-Thought (HoT) reasoning paradigm, which enables the foundation models to possess the expert-level ability of high-order multi-hop reasoning and multimodal comparative judgement. Specifically, a textual hypergraph-of-thought is constructed utilizing triple as the primary thought to model higher-order relationships, and a hyperedge-of-thought is generated through multi-hop walking paths to achieve multi-hop inference. Furthermore, we devise a visual hypergraph-of-thought to interact with the textual hypergraph-of-thought via Cross-modal Co-Attention Graph Learning for multimodal comparative verification. Experimentations on the ScienceQA benchmark demonstrate the proposed HoT-based T5 outperforms CoT-based GPT3.5 and chatGPT, which is on par with CoT-based GPT4 with a lower model size.

翻译：推理能力是基础模型最关键的能力之一，体现了其处理复杂推理任务的能力。链式思维（Chain-of-Thought, CoT）技术被广泛认为是提升基础模型推理能力的有效方法之一，并引起了广泛关注。然而，CoT的推理过程是线性、逐步的，类似于个人逻辑推理，适用于解决一般性及略微复杂的问题。相反，专家的思维模式具有两个显著特征，即高阶多跳推理和多模态比较判断，这些在CoT中无法得到妥善处理。因此，本文的核心动机是超越CoT，构建一种能够像专家一样思考的推理范式。超图的超边可以连接各种顶点，使其天然适合建模高阶关系。受此启发，本文创新性地提出了一种多模态超图思维（Hypergraph-of-Thought, HoT）推理范式，使基础模型具备专家级的高阶多跳推理和多模态比较判断能力。具体而言，我们利用三元组作为基本思维单元构建文本超图思维，以建模高阶关系，并通过多跳行走路径生成超边思维，实现多跳推理。此外，我们设计了一种视觉超图思维，通过跨模态协同注意力图学习（Cross-modal Co-Attention Graph Learning）与文本超图思维交互，实现多模态比较验证。在ScienceQA基准上的实验表明，基于HoT的T5模型性能优于基于CoT的GPT3.5和ChatGPT，且与模型规模更小的基于CoT的GPT4性能相当。