Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and February 2024. We have evaluated 9 base LLMs and 20 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model

翻译：大型语言模型（LLMs）在代码相关应用领域已成为一个突出研究方向，吸引了学术界和工业界的广泛关注。然而，随着新型和改进型LLMs的不断涌现，现有评估基准（如HumanEval、MBPP）已不足以评估其能力。本研究提出LiveCodeBench——一个面向代码的综合且无污染评估基准，该基准持续从三大竞赛平台（LeetCode、AtCoder和CodeForces）收集新近发布的编程问题。值得注意的是，本基准不仅关注代码生成能力，还聚焦于更广泛的代码相关能力，包括自我修复、代码执行和测试输出预测等。目前，LiveCodeBench收录了2023年5月至2024年2月间发布的400个高质量编程问题。我们基于LiveCodeBench对9个基础LLMs和20个指令微调LLMs进行了评估，并针对数据污染情况、整体性能对比、现有基准中的潜在过拟合问题以及各模型对比提供了实验发现。我们将发布所有提示词和模型生成结果以促进社区分析，同时提供用于添加新场景和模型评估的通用工具包。