The training of large language models (LLMs) requires substantial computational resources, complex software stacks, and carefully designed workflows to achieve scalability and efficiency. This report presents best practices and insights gained from the OpenGPT-X project, a German initiative focused on developing open, multilingual LLMs optimized for European languages. We detail the use of high-performance computing (HPC) systems, primarily JUWELS Booster at JSC, for training Teuken-7B, a 7-billion-parameter transformer model. The report covers system architecture, training infrastructure, software choices, profiling and benchmarking tools, as well as engineering and operational challenges. It includes measured throughput data of various configurations of 3D parallelism during training and the impact of features such as flash attention.

翻译：大语言模型的训练需要大量的计算资源、复杂的软件栈以及精心设计的工作流程，以实现可扩展性和效率。本报告介绍了OpenGPT-X项目（一项专注于开发针对欧洲语言优化的开放多语言大语言模型的德国倡议）中积累的最佳实践与见解。我们详细阐述了利用高性能计算系统（主要是于于利希超级计算中心的JUWELS Booster系统）训练Teuken-7B（一个70亿参数的Transformer模型）的流程。报告涵盖了系统架构、训练基础设施、软件选择、性能剖析与基准测试工具，以及工程与运维挑战。此外，还包括训练过程中不同3D并行配置的实测吞吐量数据，以及Flash Attention等特性带来的影响。