The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40\% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

翻译：BigCode社区是一个致力于负责任开发大型代码语言模型（Code LLMs）的开放科学合作组织，现推出StarCoder和StarCoderBase：参数量为155亿、支持8K上下文长度、具备填充能力且基于多查询注意力机制实现快速大批量推理的模型。StarCoderBase基于The Stack数据集（包含大量采用宽松许可协议的GitHub仓库，附带审查工具和退出机制）中的1万亿个token进行训练。我们进一步在350亿个Python token上对StarCoderBase进行微调，最终构建出StarCoder。通过迄今最全面的Code LLM评估，我们证明StarCoderBase在所有支持多编程语言的开放代码语言模型中表现最优，且性能可与OpenAI的code-cushman-001模型持平或更优。此外，StarCoder在Python微调模型中全面领先，通过提示词即可在HumanEval上实现40%的pass@1得分，同时在其他编程语言上保持原有性能。我们采取了多项重要措施以实现安全的开放访问模型发布，包括改进的个人身份信息（PII）脱敏管道和新型归因追踪工具，并基于更具商业可行性的Open Responsible AI Model许可协议公开StarCoder模型。