We investigate the feasibility of employing large language models (LLMs) for conducting the security audit of smart contracts, a traditionally time-consuming and costly process. Our research focuses on the optimization of prompt engineering for enhanced security analysis, and we evaluate the performance and accuracy of LLMs using a benchmark dataset comprising 52 Decentralized Finance (DeFi) smart contracts that have previously been compromised. Our findings reveal that, when applied to vulnerable contracts, both GPT-4 and Claude models correctly identify the vulnerability type in 40% of the cases. However, these models also demonstrate a high false positive rate, necessitating continued involvement from manual auditors. The LLMs tested outperform a random model by 20% in terms of F1-score. To ensure the integrity of our study, we conduct mutation testing on five newly developed and ostensibly secure smart contracts, into which we manually insert two and 15 vulnerabilities each. This testing yielded a remarkable best-case 78.7% true positive rate for the GPT-4-32k model. We tested both, asking the models to perform a binary classification on whether a contract is vulnerable, and a non-binary prompt. We also examined the influence of model temperature variations and context length on the LLM's performance. Despite the potential for many further enhancements, this work lays the groundwork for a more efficient and economical approach to smart contract security audits.

翻译：我们研究了采用大型语言模型（LLMs）进行智能合约安全审计的可行性，该过程传统上耗时且成本高昂。我们的研究聚焦于优化提示工程以增强安全分析，并利用包含52个此前被攻破的去中心化金融（DeFi）智能合约的基准数据集，评估了LLMs的性能和准确率。我们的发现表明，当应用于存在漏洞的合约时，GPT-4和Claude模型均能在40%的情况下正确识别漏洞类型。然而，这些模型也表现出较高的误报率，因此仍需手动审计人员持续参与。在F1分数方面，所测试的LLMs比随机模型高出20%。为确保研究的完整性，我们对五个新开发且看似安全的智能合约进行了突变测试，分别手动向其中植入2个和15个漏洞。测试结果显示，GPT-4-32k模型的最佳真阳性率高达78.7%。我们测试了两种提示方式：要求模型对合约是否存在漏洞进行二元分类，以及非二元分类提示。我们还研究了模型温度变化和上下文长度对LLM性能的影响。尽管存在许多进一步改进的潜力，但这项工作为更高效、更经济的智能合约安全审计方法奠定了基础。