Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models

翻译：大型语言模型（LLM）在推进通用人工智能（AGI）方面取得了显著进展，催生了如GPT-4和LLaMA-405B等规模不断增大的模型。然而，模型规模的扩大导致计算成本和能耗呈指数级增长，使得这些模型对于资源有限的学术研究者和企业而言并不实用。与此同时，小模型（SM）在实际应用中被频繁使用，尽管其重要性目前被低估。这引发了关于小模型在LLM时代作用的重要问题，而该主题在先前研究中受到的关注有限。在本工作中，我们从两个关键视角系统性地审视了LLM与SM之间的关系：协作与竞争。我们希望本综述能为实践者提供有价值的见解，促进对小模型贡献的更深入理解，并推动计算资源的更高效利用。代码可在https://github.com/tigerchen52/role_of_small_models获取。