We introduce the Bittensor Language Model, called "BTLM-3B-8K", a new state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and 8,192 context lengths. BTLM-3B-8K outperforms all existing 3B parameter models by 2-5.5% across downstream tasks. BTLM-3B-8K is even competitive with some 7B parameter models. Additionally, BTLM-3B-8K provides excellent long context performance, outperforming MPT-7B-8K and XGen-7B-8K on tasks up to 8,192 context length. We trained the model on a cleaned and deduplicated SlimPajama dataset; aggressively tuned the \textmu P hyperparameters and schedule; used ALiBi position embeddings; and adopted the SwiGLU nonlinearity. On Hugging Face, the most popular models have 7B parameters, indicating that users prefer the quality-size ratio of 7B models. Compacting the 7B parameter model to one with 3B parameters, with little performance impact, is an important milestone. BTLM-3B-8K needs only 3GB of memory with 4-bit precision and takes 2.5x less inference compute than 7B models, helping to open up access to a powerful language model on mobile and edge devices. BTLM-3B-8K is available under an Apache 2.0 license on Hugging Face: https://huggingface.co/cerebras/btlm-3b-8k-base.

翻译：我们介绍Bittensor语言模型"BTLM-3B-8K"，这是一款拥有30亿参数的最新一代开源语言模型。该模型基于SlimPajama数据集中的6270亿词元进行训练，并采用2048与8192两种上下文长度的混合训练策略。在下游任务中，BTLM-3B-8K以超出2-5.5%的优势全面超越现有30亿参数级模型，甚至可与部分70亿参数模型相匹敌。特别在长上下文处理方面，该模型在高达8192上下文长度的任务中表现优于MPT-7B-8K和XGen-7B-8K。我们采用经清洗去重的SlimPajama数据集进行训练；通过激进调优\textmu P超参数及训练策略；使用ALiBi位置嵌入；并采用SwiGLU非线性激活函数。在Hugging Face平台上，最受欢迎的模型多为70亿参数规模，表明用户更青睐该量级模型在质量与参数规模间的平衡。将70亿参数模型压缩至30亿参数且性能损失极小，是重要的里程碑突破。BTLM-3B-8K在4位精度下仅需3GB内存，推理计算量较70亿参数模型降低2.5倍，为移动端及边缘设备使用强大语言模型开辟了新途径。该模型基于Apache 2.0许可证发布于Hugging Face：https://huggingface.co/cerebras/btlm-3b-8k-base