In this work, we develop and release Yuan 2.0, a series of large language models with parameters ranging from 2.1 billion to 102.6 billion. The Localized Filtering-based Attention (LFA) is introduced to incorporate prior knowledge of local dependencies of natural language into Attention. A data filtering and generating system is presented to build pre-training and fine-tuning dataset in high quality. A distributed training method with non-uniform pipeline parallel, data parallel, and optimizer parallel is proposed, which greatly reduces the bandwidth requirements of intra-node communication, and achieves good performance in large-scale distributed training. Yuan 2.0 models display impressive ability in code generation, math problem-solving, and chatting compared with existing models. The latest version of YUAN 2.0, including model weights and source code, is accessible at Github.

翻译：本文开发并发布了YUAN 2.0系列大语言模型，参数量从21亿到1026亿不等。我们提出了基于局部过滤的注意力机制（LFA），将自然语言局部依赖的先验知识融入注意力机制中。同时设计了一套数据过滤与生成系统，用于构建高质量的预训练和微调数据集。在分布式训练方面，提出了一种结合非均匀流水线并行、数据并行与优化器并行的训练方法，大幅降低了节点内通信的带宽需求，并在大规模分布式训练中取得了优异性能。与现有模型相比，YUAN 2.0在代码生成、数学问题求解和对话等任务中展现出令人印象深刻的能力。YUAN 2.0的最新版本（包括模型权重和源代码）已在Github上公开发布。