With the recent advancement of Large Language Models (LLMs), generating functionally correct code has become less complicated for a wide array of developers. While using LLMs has sped up the functional development process, it poses a heavy risk to code security. Code generation with proper security measures using LLM is a significantly more challenging task than functional code generation. Security measures may include adding a pair of lines of code with the original code, consisting of null pointer checking or prepared statements for SQL injection prevention. Currently, available code repair LLMs generate code repair by supervised fine-tuning, where the model looks at cross-entropy loss. However, the original and repaired codes are mostly similar in functionality and syntactically, except for a few (1-2) lines, which act as security measures. This imbalance between the lines needed for security measures and the functional code enforces the supervised fine-tuned model to prioritize generating functional code without adding proper security measures, which also benefits the model by resulting in minimal loss. Therefore, in this work, for security hardening and strengthening of generated code from LLMs, we propose a reinforcement learning-based method for program-specific repair with the combination of semantic and syntactic reward mechanisms that focus heavily on adding security and functional measures in the code, respectively.

翻译：随着大型语言模型（LLMs）的近期进展，为各类开发者生成功能正确的代码已变得不再复杂。尽管使用LLMs加速了功能开发进程，却给代码安全带来了重大风险。利用LLM生成具备适当安全措施的代码，其难度远超功能代码生成。安全措施可能包括在原始代码中添加几行代码，例如空指针检查或用于防止SQL注入的预处理语句。目前，可用的代码修复LLM通过监督微调生成代码修复，模型基于交叉熵损失进行优化。然而，原始代码与修复后的代码在功能上和语法上大多相似，仅少数（1-2行）作为安全措施的代码存在差异。这种安全措施所需代码行与功能代码之间的不平衡，迫使监督微调后的模型优先生成功能代码，而忽略添加适当安全措施——这也使模型因损失最小化而受益。因此，在本工作中，为强化和加固LLM生成的代码安全，我们提出一种基于强化学习的方法，针对特定程序进行修复，结合语义和语法奖励机制，分别着重在代码中添加安全措施与功能保障。