Semantic understanding of programs has attracted great attention in the community. Inspired by recent successes of large language models (LLMs) in natural language understanding, tremendous progress has been made by treating programming language as another sort of natural language and training LLMs on corpora of program code. However, programs are essentially different from texts after all, in a sense that they are normally heavily structured and syntax-strict. In particular, programs and their basic units (i.e., functions and subroutines) are designed to demonstrate a variety of behaviors and/or provide possible outputs, given different inputs. The relationship between inputs and possible outputs/behaviors represents the functions/subroutines and profiles the program as a whole. Therefore, we propose to incorporate such a relationship into learning, for achieving a deeper semantic understanding of programs. To obtain inputs that are representative enough to trigger the execution of most part of the code, we resort to fuzz testing and propose fuzz tuning to boost the performance of program understanding and code representation learning, given a pre-trained LLM. The effectiveness of the proposed method is verified on two program understanding tasks including code clone detection and code classification, and it outperforms current state-of-the-arts by large margins. Code is available at https://github.com/rabbitjy/FuzzTuning.

翻译：程序语义理解一直是社区关注的焦点。受大语言模型（LLM）在自然语言理解领域取得成功的启发，研究者将编程语言视作另一种自然语言，并通过训练LLM在程序代码语料库上实现了显著进展。然而，程序本质上与文本存在根本差异：它们通常具有高度结构化和严格的语法特征。更重要的是，程序及其基本单元（即函数和子程序）被设计为根据不同输入展示多种行为或提供潜在输出。输入与可能输出/行为之间的关联不仅界定了函数/子程序的功能，更刻画了整个程序的轮廓。为此，我们提出将这种关联关系融入学习过程，以实现对程序更深层的语义理解。为获取足以触发大部分代码执行的具有代表性的输入，我们借助模糊测试技术，提出了"模糊微调"（Fuzz Tuning）方法，在预训练LLM的基础上提升程序理解与代码表示学习性能。该方法在代码克隆检测和代码分类两项程序理解任务上的有效性得到验证，其性能大幅超越现有最优方案。代码已开源至https://github.com/rabbitjy/FuzzTuning。