Large language models (LLMs) have undergone rapid evolution and achieved remarkable results in recent times. OpenAI's ChatGPT, backed by GPT-3.5 or GPT-4, has gained instant popularity due to its strong capability across a wide range of tasks, including natural language tasks, coding, mathematics, and engaging conversations. However, the impacts and limits of such LLMs in system security domain are less explored. In this paper, we delve into the limits of LLMs (i.e., ChatGPT) in seven software security applications including vulnerability detection/repair, debugging, debloating, decompilation, patching, root cause analysis, symbolic execution, and fuzzing. Our exploration reveals that ChatGPT not only excels at generating code, which is the conventional application of language models, but also demonstrates strong capability in understanding user-provided commands in natural languages, reasoning about control and data flows within programs, generating complex data structures, and even decompiling assembly code. Notably, GPT-4 showcases significant improvements over GPT-3.5 in most security tasks. Also, certain limitations of ChatGPT in security-related tasks are identified, such as its constrained ability to process long code contexts.

翻译：大语言模型（LLMs）近年来经历了快速发展，并在多个领域取得了显著成果。基于GPT-3.5或GPT-4的OpenAI的ChatGPT，凭借其在自然语言任务、编程、数学以及对话交互等广泛任务中的强大能力，迅速获得了广泛关注。然而，这类LLMs在系统安全领域的影响与极限尚未得到充分探索。本文深入研究了LLMs（即ChatGPT）在七类软件安全应用中的极限，包括漏洞检测/修复、调试、精简、反编译、补丁生成、根因分析、符号执行和模糊测试。我们的探索表明，ChatGPT不仅擅长生成代码（语言模型的传统应用），还能展现出理解用户提供的自然语言指令、推理程序中的控制流与数据流、生成复杂数据结构，甚至反编译汇编代码的强大能力。值得注意的是，在大多数安全任务中，GPT-4相比GPT-3.5展现出显著的改进。同时，我们也发现了ChatGPT在与安全相关任务中的某些局限性，例如其在处理长代码上下文方面的能力受限。