Language Models (LMs) are widely used in software engineering for code generation, but they may produce code with errors. Rather than repairing the generated code, an alternative way is to address the underlying failures of models. LM repair offers a lightweight solution to this challenge: it requires minimal data, reduces computational costs, and reduces the side effects. Unlike retraining, LM repair focuses on applying tailored updates to targeted neurons, making it ideal for scenarios with limited resources, high-performance demands, or strict safety requirements. In this paper, we propose \ul{S}emantic \ul{T}argeting for \ul{A}nalytical \ul{R}epair (\textsc{STAR}), a pioneering and novel semantic-based optimization approach for repairing LLMs. \textsc{STAR} realizes main operations in LM repair methods in an optimization process, including locating ``buggy neurons'', solving ``neuron patches'', and patching ``buggy neurons''. Correspondingly, it computes the deltas of weight matrix as the prior information to guide optimization; and attributes the targeted layers and neurons leveraging statistical insights. The neuron patches are computed with a solid semantic-based analytical formula, which directly bridges the changes to logits with the deltas of neurons, by steering latent representations. Compared to the prior work of LM repair (\textsc{MINT}) and optimization methods (\textsc{SGD}), \textsc{STAR} integrates their strengths while mitigating their limitations. \textsc{STAR} supports solving multiple failures together, significantly improving the usefulness. Evaluated on three code generation tasks using popular code LMs, \textsc{STAR} demonstrates superior effectiveness. Additionally, \textsc{STAR} exhibits better efficiency. In terms of side effects, namely the balance between generalization and specificity, \textsc{STAR} outperforms prior work by a significant margin.

翻译：语言模型（LMs）在软件工程中广泛用于代码生成，但它们可能生成包含错误的代码。除了修复生成的代码，另一种方法是解决模型的底层故障。LM修复为此挑战提供了一种轻量级解决方案：它需要的数据极少，降低了计算成本，并减少了副作用。与重新训练不同，LM修复侧重于对目标神经元应用定制化的更新，使其在资源有限、性能要求高或安全要求严格的场景中尤为理想。本文提出了一种开创性的、新颖的基于语义的优化方法用于修复大语言模型，即 \textsc{STAR}（语义定位分析修复）。\textsc{STAR} 在一个优化过程中实现了LM修复方法的主要操作，包括定位“故障神经元”、求解“神经元补丁”以及修补“故障神经元”。相应地，它计算权重矩阵的增量作为先验信息来指导优化；并利用统计洞察来归因目标层和神经元。神经元补丁通过一个坚实的基于语义的解析公式计算，该公式通过引导潜在表征，直接将logits的变化与神经元的增量联系起来。与先前的LM修复工作（\textsc{MINT}）和优化方法（\textsc{SGD}）相比，\textsc{STAR} 整合了它们的优势，同时缓解了其局限性。\textsc{STAR} 支持同时解决多个故障，显著提升了实用性。在使用流行的代码LMs对三个代码生成任务进行评估后，\textsc{STAR} 展现了卓越的有效性。此外，\textsc{STAR} 表现出更好的效率。在副作用方面，即泛化性与特异性之间的平衡，\textsc{STAR} 以显著优势超越了先前的工作。