In this paper, we present significant advancements in the pretraining of Mistral 7B, a large-scale language model, using a dataset of 32.6 GB, equivalent to 1.1 billion tokens. We explore the impact of extending the context length, releasing models with context lengths of 4096 and 32768 tokens, and further refining performance with a specialized 16384 context length instruction-tuned model, we called it Malaysian Mistral. Our experiments demonstrate the efficacy of continue pretraining and the influence of extended context lengths on Mistral 7B's language understanding capabilities. Additionally, we release a model specifically tuned with a 16384 context length instruction, showcasing its potential for capturing nuanced language intricacies. Furthermore, our research contributes to the benchmarking of Malaysian Mistral against prominent language models, including ChatGPT3.5 and Claude 2. We present compelling results indicating Malaysian Mistral's superior performance on Tatabahasa (Malay grammar) test set, particularly when fine-tuned with instructions. All models released at https://huggingface.co/collections/mesolitica/malaysian-mistral-7b-6528f2ec825f4bba46c1700c

翻译：本文介绍了我们在基于Mistral 7B的大规模语言模型预训练方面取得的重要进展，使用了32.6 GB（相当于11亿个令牌）的数据集。我们研究了扩展上下文长度的影响，发布了上下文长度为4096和32768个令牌的模型，并通过专门优化的16384上下文长度指令调优模型（我们称之为Malaysian Mistral）进一步提升了性能。实验证明了持续预训练的有效性以及扩展上下文长度对Mistral 7B语言理解能力的影响。此外，我们还发布了一个专门针对16384上下文长度指令调优的模型，展示了其在捕捉细微语言特征方面的潜力。进一步研究中，我们针对Malaysian Mistral与包括ChatGPT3.5和Claude 2在内的主流语言模型进行了基准测试。结果显示，Malaysian Mistral在马来语语法（Tatabahasa）测试集上表现更优，尤其是在经过指令微调后。所有模型已发布于https://huggingface.co/collections/mesolitica/malaysian-mistral-7b-6528f2ec825f4bba46c1700c。