With the recent advancement of Large Language Models (LLMs), generating functionally correct code has become less complicated for a wide array of developers. While using LLMs has sped up the functional development process, it poses a heavy risk to code security. Code generation with proper security measures using LLM is a significantly more challenging task than functional code generation. Security measures may include adding a pair of lines of code with the original code, consisting of null pointer checking or prepared statements for SQL injection prevention. Currently, available code repair LLMs generate code repair by supervised fine-tuning, where the model looks at cross-entropy loss. However, the original and repaired codes are mostly similar in functionality and syntactically, except for a few (1-2) lines, which act as security measures. This imbalance between the lines needed for security measures and the functional code enforces the supervised fine-tuned model to prioritize generating functional code without adding proper security measures, which also benefits the model by resulting in minimal loss. Therefore, in this work, for security hardening and strengthening of generated code from LLMs, we propose a reinforcement learning-based method for program-specific repair with the combination of semantic and syntactic reward mechanisms that focus heavily on adding security and functional measures in the code, respectively.

翻译：随着大语言模型（LLM）的最新进展，为各类开发者生成功能正确的代码已变得不再复杂。虽然使用LLM加速了功能开发过程，但这给代码安全带来了重大风险。使用LLM生成具备适当安全措施的代码，远比生成功能性代码更具挑战性。安全措施可能包括在原始代码中添加一两行代码，例如空指针检查或用于防止SQL注入的预处理语句。目前，可用的代码修复LLM通过监督微调生成代码修复，该模型关注交叉熵损失。然而，原始代码与修复后的代码在功能和语法上大多相似，仅有少数（1-2行）代码行作为安全措施。这种安全措施所需代码行与功能性代码之间的不平衡，迫使经过监督微调的模型优先生成功能性代码，而未能添加适当的安全措施，这也有助于模型实现最小化损失。因此，在本研究中，为了对LLM生成的代码进行安全加固与强化，我们提出了一种基于强化学习的程序特定修复方法，结合了语义和语法奖励机制，分别着重于在代码中添加安全和功能措施。