The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at https://hf.co/bigcode.

翻译：BigCode项目是一项致力于负责任开发代码大语言模型的开源科学合作。本技术报告描述了截至2022年12月的合作进展，概述了个人身份信息(PII)脱敏管道的当前状态、为降低模型架构风险而开展的实验，以及探索更优训练数据预处理方法的实验。我们在The Stack数据集的Java、JavaScript和Python子集上训练了1.1B参数模型，并在MultiPL-E文本到代码基准上进行评估。研究发现，对近似重复数据采取更激进的过滤策略能进一步提升性能，且令人意外的是，从拥有5个以上GitHub星标的仓库中选择文件会显著降低性能。尽管我们的最佳模型规模显著更小，但它在MultiPL-E的Java、JavaScript和Python部分，无论是从左到右生成还是代码填充任务中，均优于此前开源的跨语言代码生成模型（InCoder-6.7B和CodeGen-Multi-2.7B）。所有模型均采用OpenRAIL许可证发布，地址为：https://hf.co/bigcode。