The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.

翻译：大型语言模型（LLMs）领域蓬勃发展，以OpenAI的ChatGPT等先进模型为代表，标志着人工智能的重大进步。然而，这些模型在计算、内存、能源和财务资源方面带来了高消耗的巨大挑战，尤其是在资源受限的环境中。本综述旨在系统性地应对这些挑战，通过回顾一系列旨在提升LLM资源效率的技术。我们根据优化重点对这些方法进行分类：计算、内存、能源、财务和网络资源，并考察其在LLM生命周期各阶段（包括架构设计、预训练、微调和系统设计）的适用性。此外，本综述还基于具体资源类型对资源效率技术进行了精细分类，揭示了各类资源与相应优化技术之间的复杂关系与映射。我们还提出了一套标准化的评估指标与数据集，以促进不同模型与技术间的一致且公平的比较。通过全面概述当前最优方法并识别开放研究途径，本综述为研究人员与实践者提供了基础性参考，助力他们在快速发展的格局中开发更可持续、更高效的LLM。