The rapid development of deep learning techniques, improved computational power, and the availability of vast training data have led to significant advancements in pre-trained models and large language models (LLMs). Pre-trained models based on architectures such as BERT and Transformer, as well as LLMs like ChatGPT, have demonstrated remarkable language capabilities and found applications in Software engineering. Software engineering tasks can be divided into many categories, among which generative tasks are the most concern by researchers, where pre-trained models and LLMs possess powerful language representation and contextual awareness capabilities, enabling them to leverage diverse training data and adapt to generative tasks through fine-tuning, transfer learning, and prompt engineering. These advantages make them effective tools in generative tasks and have demonstrated excellent performance. In this paper, we present a comprehensive literature review of generative tasks in SE using pre-trained models and LLMs. We accurately categorize SE generative tasks based on software engineering methodologies and summarize the advanced pre-trained models and LLMs involved, as well as the datasets and evaluation metrics used. Additionally, we identify key strengths, weaknesses, and gaps in existing approaches, and propose potential research directions. This review aims to provide researchers and practitioners with an in-depth analysis and guidance on the application of pre-trained models and LLMs in generative tasks within SE.

翻译：深度学习技术的快速发展、计算能力的提升以及海量训练数据的可用性，推动了预训练模型和大规模语言模型（LLMs）的重大进展。基于BERT和Transformer等架构的预训练模型，以及ChatGPT等LLMs，已展现出卓越的语言能力，并在软件工程领域得到应用。软件工程任务可分为多个类别，其中生成式任务最受研究者关注。预训练模型和LLMs具备强大的语言表示和上下文感知能力，能够利用多样化的训练数据，并通过微调、迁移学习和提示工程适应生成式任务。这些优势使其成为生成式任务中的有效工具，并表现出优异的性能。本文对使用预训练模型和LLMs的软件工程生成式任务进行了全面的文献综述。我们基于软件工程方法论对生成式任务进行了精确分类，总结了所涉及的先进预训练模型和LLMs，以及使用的数据集和评估指标。此外，我们识别了现有方法的关键优势、不足和空白，并提出了潜在的研究方向。本综述旨在为研究者和实践者提供关于预训练模型和LLMs在软件工程生成式任务中应用的深入分析与指导。