Over the past year, spatial intelligence has drawn increasing attention. Many prior works study it from the perspective of visual-spatial intelligence, where models have access to visuospatial information from visual inputs. However, in the absence of visual information, whether linguistic intelligence alone is sufficient to endow models with spatial intelligence, and how models perform relevant tasks with text-only inputs still remain unexplored. Therefore, in this paper, we focus on a fundamental and critical capability in spatial intelligence from a linguistic perspective: viewpoint rotation understanding (VRU). Specifically, LLMs and VLMs are asked to infer their final viewpoint and predict the corresponding observation in an environment given textual description of viewpoint rotation and observation over multiple steps. We find that both LLMs and VLMs perform poorly on our proposed dataset while human can easily achieve 100% accuracy, indicating a substantial gap between current model capabilities and the requirements of spatial intelligence. To uncover the underlying mechanisms, we conduct a layer-wise probing analysis and head-wise causal intervention. Our findings reveal that although models encode viewpoint information in the hidden states, they appear to struggle to bind the viewpoint position with corresponding observation, resulting in a hallucination in final layers. Finally, we selectively fine-tune the key attention heads identified by causal intervention to improve VRU performance. Experimental results demonstrate that such selective fine-tuning achieves improved VRU performance while avoiding catastrophic forgetting of generic abilities. Our dataset and code will be released at https://github.com/Young-Zhen/VRU_Interpret .

翻译：在过去一年中，空间智能受到了越来越多的关注。许多先前的研究从视觉-空间智能的角度展开，模型能够从视觉输入中获取视觉空间信息。然而，在缺乏视觉信息的情况下，仅凭语言智能是否足以赋予模型空间智能，以及模型如何仅通过文本输入执行相关任务，仍是一个未被探索的问题。因此，本文从语言学的角度聚焦于空间智能中一项基础且关键的能力：视角旋转理解（VRU）。具体而言，要求大语言模型（LLMs）和多模态大模型（VLMs）在多步操作中，根据给定的视角旋转与观测的文本描述，推断最终视角并预测对应的环境观测结果。我们发现，LLMs和VLMs在我们提出的数据集上表现较差，而人类可以轻松达到100%的准确率，这表明当前模型能力与空间智能要求之间存在显著差距。为了揭示其内在机制，我们进行了逐层探针分析和逐头因果干预。研究发现，尽管模型在隐状态中编码了视角信息，但似乎难以将视角位置与对应的观测结果绑定，导致最终层出现幻觉。最后，我们选择性地微调了因果干预识别的关键注意力头，以提升VRU性能。实验结果表明，这种选择性微调在提升VRU性能的同时，避免了通用能力的灾难性遗忘。我们的数据集和代码将发布在 https://github.com/Young-Zhen/VRU_Interpret。