Auto-completing code enables developers to speed up coding significantly. Recent advances in transformer-based large language model (LLM) technologies have been applied to code synthesis. However, studies show that many of such synthesized codes contain vulnerabilities. We propose a novel vulnerability-constrained decoding approach to reduce the amount of vulnerable code generated by such models. Using a small dataset of labeled vulnerable lines of code, we fine-tune an LLM to include vulnerability labels when generating code, acting as an embedded classifier. Then, during decoding, we deny the model to generate these labels to avoid generating vulnerable code. To evaluate the method, we chose to automatically complete Ethereum Blockchain smart contracts (SCs) as the case study due to the strict requirements of SC security. We first fine-tuned the 6-billion-parameter GPT-J model using 186,397 Ethereum SCs after removing the duplication from 2,217,692 SCs. The fine-tuning took more than one week using ten GPUs. The results showed that our fine-tuned model could synthesize SCs with an average BLEU (BiLingual Evaluation Understudy) score of 0.557. However, many codes in the auto-completed SCs were vulnerable. Using the code before the vulnerable line of 176 SCs containing different types of vulnerabilities to auto-complete the code, we found that more than 70% of the auto-completed codes were insecure. Thus, we further fine-tuned the model on other 941 vulnerable SCs containing the same types of vulnerabilities and applied vulnerability-constrained decoding. The fine-tuning took only one hour with four GPUs. We then auto-completed the 176 SCs again and found that our approach could identify 62% of the code to be generated as vulnerable and avoid generating 67% of them, indicating the approach could efficiently and effectively avoid vulnerabilities in the auto-completed code.

翻译：自动补全代码能够显著提升开发者的编码速度。基于Transformer的大语言模型（LLM）技术的最新进展已被应用于代码合成领域。然而研究表明，许多此类合成代码存在安全漏洞。我们提出了一种新颖的漏洞约束解码方法，用于减少此类模型生成的易受攻击代码量。通过使用少量标注过漏洞行的代码数据集，我们对大语言模型进行微调，使其在生成代码时包含漏洞标签作为嵌入式分类器。随后在解码过程中，我们禁止模型生成这些标签，从而避免生成易受攻击的代码。为评估该方法，我们选择以太坊区块链智能合约（SC）的自动补全作为案例研究，因其对安全性要求极为严格。我们首先从2217692个以太坊智能合约中剔除重复项后，使用186397个合约对具有60亿参数的GPT-J模型进行微调。该微调过程使用十块GPU耗时超过一周。结果显示，微调后的模型合成智能合约的平均BLEU（双语评估替补）得分为0.557。然而自动补全的智能合约中仍存在大量易受攻击代码。我们选取了176个包含不同类型漏洞的智能合约，利用漏洞行之前的代码进行自动补全，发现超过70%的自动补全代码存在安全隐患。为此，我们进一步在另外941个包含相同类型漏洞的智能合约上对模型进行微调，并应用漏洞约束解码。此次微调使用四块GPU仅耗时一小时。随后我们重新对这176个智能合约进行自动补全，发现该方法能识别出62%待生成的易受攻击代码，并成功避免其中67%的漏洞生成，表明该方法能够高效且有效地避免自动补全代码中的漏洞。