Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs). However, applying LLMs to graphs requires linearizing their structure into sequences, introducing distortion rooted in the graph bandwidth problem. While this distortion has been shown to degrade performance, it is often attributed to prompt design or model scale, leaving the underlying mechanism unclear. In this work, we show \textit{how} rotary positional embeddings turn graph linearization into bandwidth-dependent attention decay, suppressing attention between graph-adjacent nodes that are forced far apart in the serialized sequence. This shifts the focus of LLM-based graph reasoning from prompt engineering and scaling toward correcting attention misalignment. Motivated by this analysis, we propose \textbf{G}raph-\textbf{a}ligned \textbf{L}anguage \textbf{A}ttention (\textbf{GaLA}), a lightweight, inference-time modification for LLMs. GaLA biases attention toward graph-adjacent nodes while preserving the LLM's sequential inductive biases. Across TAG benchmarks, GaLA improves performance with negligible overhead, demonstrating that distortion is a correctable bottleneck in LLM-based graph reasoning.

翻译：大语言模型（LLM）在基于文本属性图（TAG）的推理任务中展现出潜力。然而，将LLM应用于图结构需要将其结构线性化为序列，这引入了源于图带宽问题的失真。尽管这种失真已被证实会降低性能，但过往研究常将其归因于提示设计或模型规模，其潜在机制尚不明确。本工作揭示了旋转位置嵌入如何将图线性化转化为依赖带宽的注意力衰减，抑制了序列化序列中因被迫分隔过远的图邻接节点间的注意力交互，从而将LLM图推理的研究焦点从提示工程和规模扩展转向修正注意力偏移。基于此分析，我们提出**图对齐语言注意力（GaLA）**——一种轻量级、推理时生效的LLM修改方法。GaLA在保持LLM序列归纳偏好的同时，将注意力偏向图邻接节点。在多个TAG基准测试中，GaLA以可忽略的额外开销提升了性能，表明失真是基于LLM图推理中的一个可修正瓶颈。