Large language models (LLMs) have significantly advanced the field of automated code generation. However, a notable research gap exists in the evaluation of social biases that may be present in the code produced by LLMs. To solve this issue, we propose a novel fairness framework, i.e., Solar, to assess and mitigate the social biases of LLM-generated code. Specifically, Solar can automatically generate test cases for quantitatively uncovering social biases of the auto-generated code by LLMs. To quantify the severity of social biases in generated code, we develop a dataset that covers a diverse set of social problems. We applied Solar and the crafted dataset to four state-of-the-art LLMs for code generation. Our evaluation reveals severe bias in the LLM-generated code from all the subject LLMs. Furthermore, we explore several strategies for bias mitigation, including Chain-of-Thought (CoT) prompting, combining positive role-playing with CoT prompting and iterative prompting. Our experiments show that iterative prompting can effectively reduce social bias in LLM-generated code by up to 90%. Solar is highly extensible to evaluate new social problems.

翻译：大语言模型（LLMs）极大地推动了自动化代码生成领域的发展。然而，在评估LLMs生成代码中可能存在的社会偏见方面，仍存在显著的研究空白。为解决这一问题，我们提出了一种新颖的公平性评估框架——Solar，用于评估并缓解LLM生成代码中的社会偏见。具体而言，Solar能够自动生成测试用例，以定量揭示LLMs自动生成代码中的社会偏见。为量化生成代码中社会偏见的严重程度，我们构建了一个涵盖多样化社会问题的数据集。我们将Solar及所构建的数据集应用于四种最先进的代码生成LLMs。评估结果表明，所有受测LLMs生成的代码均存在严重的偏见。此外，我们探索了多种偏见缓解策略，包括思维链（CoT）提示、结合积极角色扮演与CoT提示以及迭代式提示。实验表明，迭代式提示能有效将LLM生成代码中的社会偏见降低高达90%。Solar框架具备高度可扩展性，可用于评估新的社会问题。