Recently, high-performing code generation systems based on large language models have surfaced. They are trained on massive corpora containing much more natural text than actual executable computer code. This work shows that current code generation systems exhibit undesired biases inherited from their large language model backbones, which can reduce the quality of the generated code under specific circumstances. To investigate the effect, we propose the "block of influence" concept, which enables a modular decomposition and analysis of the coding challenges. We introduce an automated intervention mechanism reminiscent of adversarial testing that exposes undesired biases through the failure modes of the models under test. Finally, we demonstrate how our framework can be used as a data transformation technique during fine-tuning, acting as a mitigation strategy for these biases.

翻译：近期，基于大语言模型的高性能代码生成系统不断涌现。这些系统在包含大量自然文本（远超实际可执行计算机代码）的海量语料库上进行训练。本研究表明，当前代码生成系统存在从其大语言模型基础架构继承而来的不良偏见，这种偏见在特定情境下会降低生成代码的质量。为探究这一影响，我们提出"影响区块"概念，支持对编程挑战进行模块化解构与分析。我们引入一种类似对抗性测试的自动干预机制，通过被测模型的故障模式暴露其隐藏的偏见。最后，我们论证了该框架如何在微调阶段作为数据转换技术，从而成为缓解这些偏见的一种有效策略。