As models continue to grow in size, the development of memory optimization methods (MOMs) has emerged as a solution to address the memory bottleneck encountered when training large models. To comprehensively examine the practical value of various MOMs, we have conducted a thorough analysis of existing literature from a systems perspective. Our analysis has revealed a notable challenge within the research community: the absence of standardized metrics for effectively evaluating the efficacy of MOMs. The scarcity of informative evaluation metrics hinders the ability of researchers and practitioners to compare and benchmark different approaches reliably. Consequently, drawing definitive conclusions and making informed decisions regarding the selection and application of MOMs becomes a challenging endeavor. To address the challenge, this paper summarizes the scenarios in which MOMs prove advantageous for model training. We propose the use of distinct evaluation metrics under different scenarios. By employing these metrics, we evaluate the prevailing MOMs and find that their benefits are not universal. We present insights derived from experiments and discuss the circumstances in which they can be advantageous.

翻译：随着模型规模持续扩大，内存优化方法的发展已成为解决大规模模型训练中内存瓶颈问题的有效方案。为全面评估各类内存优化方法的实际价值，我们从系统视角对现有文献进行了深入分析。分析发现研究领域存在一个显著挑战：缺乏标准化指标体系来有效评估内存优化方法的效能。信息性评估指标的匮乏阻碍了研究人员和实践者对不同方法进行可靠对比与基准测试，由此导致在方法选择与应用方面难以得出确定性结论和做出明智决策。针对这一挑战，本文归纳了内存优化方法对模型训练具有优势的应用场景，并提出在不同场景下采用差异化评估指标。通过应用这些指标，我们对主流内存优化方法进行评估后发现其收益并非普适。我们呈现了实验所得见解，并探讨了这些方法能够发挥优势的具体情况。