Integrating reasoning in large language models and large vision-language models has recently led to significant improvement of their capabilities. However, the generalization of reasoning models is still vaguely defined and poorly understood. In this work, we present an evaluation framework to rigorously examine how well chain-of-thought (CoT) approaches generalize on a simple planning task. Specifically, we consider a grid-based navigation task in which a model is provided with a map and must output a sequence of moves that guides a player from a start position to a goal while avoiding obstacles. The versatility of the task and its data allows us to fine-tune model variants using different input representations (visual and textual) and CoT reasoning strategies, and systematically evaluate them under both in-distribution (ID) and out-of-distribution (OOD) test conditions. Our experiments show that, while CoT reasoning improves in-distribution generalization across all representations, out-of-distribution generalization (e.g., to larger maps) remains very limited in most cases when controlling for trivial matches with the ID data. Surprisingly, we find that reasoning traces which combine multiple text formats yield the best (and non-trivial) OOD generalization. Finally, purely text-based models consistently outperform those utilizing image-based inputs, including a recently proposed approach relying on latent space reasoning.

翻译：近年来，将推理能力整合到大型语言模型与大型视觉语言模型中，显著提升了这些模型的能力。然而，推理模型的泛化能力仍定义模糊且缺乏深入理解。在本研究中，我们提出了一个评估框架，以严格检验思维链方法在简单规划任务上的泛化表现。具体而言，我们考虑一个基于网格的导航任务：模型接收一张地图，必须输出一系列移动指令，引导玩家从起点位置到达目标位置，同时避开障碍物。该任务及其数据的多样性使我们能够利用不同的输入表示（视觉与文本）和思维链推理策略对模型变体进行微调，并在分布内与分布外测试条件下对其进行系统评估。我们的实验表明，尽管思维链推理提升了所有表示形式下的分布内泛化能力，但在控制与分布内数据的简单匹配后，分布外泛化（例如到更大尺寸的地图）在大多数情况下仍然非常有限。令人惊讶的是，我们发现结合多种文本格式的推理轨迹能实现最佳（且非平凡的）分布外泛化。最后，纯文本模型在各项评估中始终优于基于图像输入的模型，包括近期提出的一种依赖潜在空间推理的方法。