This paper tackles the challenge of teaching code semantics to Large Language Models (LLMs) for program analysis by incorporating code symmetries into the model architecture. We introduce a group-theoretic framework that defines code symmetries as semantics-preserving transformations, where forming a code symmetry group enables precise and efficient reasoning of code semantics. Our solution, SymC, develops a novel variant of self-attention that is provably equivariant to code symmetries from the permutation group defined over the program dependence graph. SymC obtains superior performance on five program analysis tasks, outperforming state-of-the-art code models, including GPT-4, without any pre-training. Our results suggest that code LLMs that encode the code structural prior via the code symmetry group generalize better and faster.

翻译：本文通过将代码对称性融入模型架构，解决了大型语言模型（LLM）在程序分析中学习代码语义的挑战。我们提出一个基于群论的框架，将代码对称性定义为保持语义不变的变换，形成代码对称群后能对代码语义进行精确高效的推理。我们的解决方案SymC开发了一种新型自注意力机制，该机制对定义在程序依赖图上的置换群所对应的代码对称性具有可证明的等变性。SymC在五项程序分析任务中取得了卓越性能，无需任何预训练即超越了包括GPT-4在内的最新代码模型。实验结果表明，通过代码对称群编码代码结构先验知识的代码LLM具有更优且更快的泛化能力。