Foundation models (e.g., ChatGPT, DALL-E, PengCheng Mind, PanGu-$\Sigma$) have demonstrated extraordinary performance in key technological areas, such as natural language processing and visual recognition, and have become the mainstream trend of artificial general intelligence. This has led more and more major technology giants to dedicate significant human and financial resources to actively develop their foundation model systems, which drives continuous growth of these models' parameters. As a result, the training and serving of these models have posed significant challenges, including substantial computing power, memory consumption, bandwidth demands, etc. Therefore, employing efficient training and serving strategies becomes particularly crucial. Many researchers have actively explored and proposed effective methods. So, a comprehensive survey of them is essential for system developers and researchers. This paper extensively explores the methods employed in training and serving foundation models from various perspectives. It provides a detailed categorization of these state-of-the-art methods, including finer aspects such as network, computing, and storage. Additionally, the paper summarizes the challenges and presents a perspective on the future development direction of foundation model systems. Through comprehensive discussion and analysis, it hopes to provide a solid theoretical basis and practical guidance for future research and applications, promoting continuous innovation and development in foundation model systems.

翻译：基础模型（如ChatGPT、DALL-E、鹏城脑海、盘古-Σ）在自然语言处理、视觉识别等关键技术领域展现出卓越性能，已成为通用人工智能的主流发展趋势。这使得越来越多的大型科技企业投入大量人力和资金积极开发基础模型系统，推动这些模型的参数规模持续增长。因此，这些模型的训练与部署面临重大挑战，包括巨大的算力需求、内存消耗、带宽要求等。采用高效的训练与服务策略变得尤为关键。众多研究者已积极开展探索并提出有效方法。对此进行全面综述对系统开发者和研究人员至关重要。本文从多角度深入探讨基础模型训练与服务的各类方法，对这些前沿方法进行细致分类，涵盖网络、计算、存储等更细粒度层面。同时，本文总结了现有挑战，并展望了基础模型系统的未来发展方向。通过全面的讨论与分析，本文旨在为未来研究与应用提供坚实的理论基础和实践指导，推动基础模型系统的持续创新与发展。