Evaluating the alignment of large language models (LLMs) with user-defined coding preferences is a challenging endeavour that requires a deep assessment of LLMs' outputs. Existing methods and benchmarks rely primarily on automated metrics and static analysis tools, which often fail to capture the nuances of user instructions and LLM outputs. To address this gap, we propose using the LLM-as-a-Judge methodology to evaluate the alignment of LLMs with coding preferences. Based on this approach, we present CodeUltraFeedback, a comprehensive dataset designed to facilitate the evaluation and improvement of LLM alignment. CodeUltraFeedback consists of 10,000 coding instructions, each annotated with four responses generated from a diverse pool of 14 LLMs. These responses are ranked based on five distinct coding preferences using GPT-3.5 as a judge, providing both numerical scores and detailed textual feedback. Our analysis of CodeUltraFeedback reveals that responses from GPT-3.5 and GPT-4 are generally preferred over those from open-weight LLMs, highlighting significant differences in alignment between closed and open-weight models. In turn, we explore the usage of CodeUltraFeedback as feedback data to fine-tune and align CodeLlama-7B-Instruct using supervised fine-tuning (SFT) and reinforcement learning from AI feedback (RLAIF) with direct preference optimization (DPO). The resulting aligned CodeLlama-7B-Instruct model outperforms larger LLMs in terms of alignment with coding preferences and shows improved functional correctness on the HumanEval+ benchmark compared to the original instruct model. Therefore, our contributions bridge the gap in preference tuning of LLMs for code and set the stage for further advancements in model alignment and RLAIF in automated software engineering.

翻译：评估大型语言模型（LLMs）与用户定义编码偏好之间的对齐是一项具有挑战性的任务，需要对LLMs的输出进行深入评估。现有方法和基准主要依赖自动化指标和静态分析工具，这些工具往往无法捕捉用户指令和LLM输出的细微差别。为弥补这一不足，我们提出采用LLM-as-a-Judge方法来评估LLMs与编码偏好的对齐程度。基于此方法，我们提出了CodeUltraFeedback——一个旨在促进LLM对齐评估与改进的综合数据集。CodeUltraFeedback包含10,000条编码指令，每条指令均标注有从14个不同LLMs生成的四个响应。这些响应使用GPT-3.5作为评判者，依据五种不同的编码偏好进行排序，并提供数值评分和详细的文本反馈。我们对CodeUltraFeedback的分析表明，GPT-3.5和GPT-4生成的响应通常优于开源权重LLMs的响应，这凸显了闭源模型与开源权重模型在对齐性上的显著差异。进一步地，我们探索了将CodeUltraFeedback作为反馈数据，通过监督微调（SFT）和基于AI反馈的强化学习（RLAIF）结合直接偏好优化（DPO），对CodeLlama-7B-Instruct进行微调和对齐。最终得到的对齐后CodeLlama-7B-Instruct模型在编码偏好对齐方面优于更大的LLMs，并且在HumanEval+基准测试中相比原始指令模型展现出更强的功能正确性。因此，我们的工作弥补了代码领域LLM偏好调优的空白，并为自动化软件工程中的模型对齐和RLAIF技术的进一步发展奠定了基础。