Deep Learning (DL) library bugs affect downstream DL applications, emphasizing the need for reliable systems. Generating valid input programs for fuzzing DL libraries is challenging due to the need for satisfying both language syntax/semantics and constraints for constructing valid computational graphs. Recently, the TitanFuzz work demonstrates that modern Large Language Models (LLMs) can be directly leveraged to implicitly learn all the constraints to generate valid DL programs for fuzzing. However, LLMs tend to generate ordinary programs following similar patterns seen in their massive training corpora, while fuzzing favors unusual inputs that cover edge cases or are unlikely to be manually produced. To fill this gap, this paper proposes FuzzGPT, the first technique to prime LLMs to synthesize unusual programs for fuzzing. FuzzGPT is built on the well-known hypothesis that historical bug-triggering programs may include rare/valuable code ingredients important for bug finding. Traditional techniques leveraging such historical information require intensive human efforts to design dedicated generators and ensure the validity of generated programs. FuzzGPT demonstrates that this process can be fully automated via the intrinsic capabilities of LLMs (including fine-tuning and in-context learning), while being generalizable and applicable to challenging domains. While FuzzGPT can be applied with different LLMs, this paper focuses on the powerful GPT-style models: Codex and CodeGen. Moreover, FuzzGPT also shows the potential of directly leveraging the instruct-following capability of the recent ChatGPT for effective fuzzing. Evaluation on two popular DL libraries (PyTorch and TensorFlow) shows that FuzzGPT can substantially outperform TitanFuzz, detecting 76 bugs, with 49 already confirmed as previously unknown bugs, including 11 high-priority bugs or security vulnerabilities.

翻译：深度学习（DL）库的缺陷会波及下游DL应用，凸显了系统可靠性的重要性。在为DL库生成有效的模糊测试输入程序时，既需要满足语言语法/语义，又需满足构建有效计算图的约束条件，这极具挑战性。近年来，TitanFuzz研究表明，可直接利用现代大型语言模型（LLMs）隐式学习所有约束条件，从而生成有效的DL模糊测试程序。然而，LLMs倾向于生成与其大规模训练语料中相似模式的常规程序，而模糊测试则更青睐能覆盖边缘案例或难以手动生成的非常规输入。为填补这一空白，本文提出FuzzGPT——首个引导LLMs合成非常规模糊测试程序的技术。FuzzGPT基于一个广为人知的假设：历史触发缺陷的程序可能包含对缺陷发现至关重要的罕见/有价值代码片段。传统利用此类历史信息的技术需要大量人工投入设计专用生成器，并确保生成程序的有效性。FuzzGPT表明，借助LLMs的内在能力（包括微调和上下文学习），该过程可完全自动化，同时具备泛化性和对挑战性领域的适用性。尽管FuzzGPT可适配不同LLMs，本文聚焦于强大的GPT风格模型：Codex和CodeGen。此外，FuzzGPT还展示了直接利用近期ChatGPT的指令跟随能力进行高效模糊测试的潜力。在两大流行DL库（PyTorch和TensorFlow）上的评估表明，FuzzGPT可显著优于TitanFuzz，检测到76个缺陷，其中49个已被确认为此前未知的缺陷，包括11个高优先级缺陷或安全漏洞。