Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

翻译：语言本质上是一个由语法规则支配的复杂精巧的人类表达系统。开发能够理解和掌握语言的人工智能算法是一项重大挑战。作为一种主要方法，语言建模在过去二十年中被广泛研究，用于语言理解和生成，从统计语言模型发展到神经语言模型。近年来，通过在大规模语料库上预训练Transformer模型，提出了预训练语言模型（PLMs），这些模型在解决各种自然语言处理任务中展现出强大能力。由于研究人员发现模型扩展可以带来性能提升，他们通过将模型规模扩大到更大尺寸进一步研究缩放效应。有趣的是，当参数规模超过一定水平时，这些扩大的语言模型不仅实现了显著的性能提升，还展现出一些小型语言模型不具备的特殊能力。为了区分参数规模的差异，研究界将这种大规模PLMs命名为"大语言模型（LLM）"。近年来，学术界和工业界都大幅推进了大语言模型的研究，其中一项显著进展是ChatGPT的发布，引发了社会各界的广泛关注。大语言模型的技术演进正在对整个AI社区产生重要影响，这将彻底改变我们开发和利用AI算法的方式。在本综述中，我们通过介绍背景、关键发现和主流技术，回顾了大语言模型的最新进展。特别地，我们重点关注大语言模型的四个主要方面，即预训练、适配调优、利用和能力评估。此外，我们还总结了开发大语言模型的可用资源，并讨论了未来方向中尚待解决的问题。