We investigate the feasibility of employing large language models (LLMs) for conducting the security audit of smart contracts, a traditionally time-consuming and costly process. Our research focuses on the optimization of prompt engineering for enhanced security analysis, and we evaluate the performance and accuracy of LLMs using a benchmark dataset comprising 52 Decentralized Finance (DeFi) smart contracts that have previously been compromised. Our findings reveal that, when applied to vulnerable contracts, both GPT-4 and Claude models correctly identify the vulnerability type in 40% of the cases. However, these models also demonstrate a high false positive rate, necessitating continued involvement from manual auditors. The LLMs tested outperform a random model by 20% in terms of F1-score. To ensure the integrity of our study, we conduct mutation testing on five newly developed and ostensibly secure smart contracts, into which we manually insert two and 15 vulnerabilities each. This testing yielded a remarkable best-case 78.7% true positive rate for the GPT-4-32k model. We tested both, asking the models to perform a binary classification on whether a contract is vulnerable, and a non-binary prompt. We also examined the influence of model temperature variations and context length on the LLM's performance. Despite the potential for many further enhancements, this work lays the groundwork for a more efficient and economical approach to smart contract security audits.

翻译：我们研究了使用大型语言模型（LLMs）进行智能合约安全审计的可行性，而传统上这是一个耗时且成本高昂的过程。我们的研究聚焦于优化提示工程以增强安全分析，并利用一个包含52个先前被攻破的去中心化金融（DeFi）智能合约的基准数据集，评估了LLMs的性能和准确率。研究结果表明，当应用于存在漏洞的合约时，GPT-4和Claude模型均能在40%的情况下正确识别漏洞类型。然而，这些模型也表现出较高的误报率，因此仍需要人工审计师的持续参与。经过测试的LLMs在F1分数上比随机模型高出20%。为确保研究的完整性，我们对五个新开发且表面安全的智能合约进行了突变测试，分别手动向每个合约中插入两个和十五个漏洞。测试结果显示，GPT-4-32k模型取得了最佳情况下高达78.7%的真阳性率。我们测试了两种提示方式：要求模型对合约是否易受攻击进行二元分类，以及非二元提示。我们还研究了模型温度变化和上下文长度对LLM性能的影响。尽管仍有进一步改进的潜力，但这项工作为更高效、更经济的智能合约安全审计方法奠定了基础。