As deep learning models exponentially increase in size, optimizers such as Adam encounter significant memory consumption challenges due to the storage of first and second moment data. Current memory-efficient methods like Adafactor and CAME often compromise accuracy with their matrix factorization techniques. Addressing this, we introduce Adapprox, a novel approach that employs randomized low-rank matrix approximation for a more effective and accurate approximation of Adam's second moment. Adapprox features an adaptive rank selection mechanism, finely balancing accuracy and memory efficiency, and includes an optional cosine similarity guidance strategy to enhance stability and expedite convergence. In GPT-2 training and downstream tasks, Adapprox surpasses AdamW by achieving 34.5% to 49.9% and 33.8% to 49.9% memory savings for the 117M and 345M models, respectively, with the first moment enabled, and further increases these savings without the first moment. Besides, it enhances convergence speed and improves downstream task performance relative to its counterparts.

翻译：摘要：随着深度学习模型规模呈指数级增长，Adam等优化器因需存储一阶矩和二阶矩数据而面临显著的内存消耗挑战。当前诸如Adafactor和CAME等内存高效方法因采用矩阵分解技术往往牺牲准确性。为解决此问题，我们提出Adapprox——一种通过随机低秩矩阵近似对Adam二阶矩进行更有效且精准近似的创新方法。Adapprox具备自适应秩选择机制，可精细平衡精度与内存效率，并包含一个可选的余弦相似度引导策略以增强稳定性并加速收敛。在GPT-2训练及下游任务中，Adapprox在启用一阶矩时分别为117M和345M模型节省34.5%至49.9%及33.8%至49.9%的显存，且在不启用一阶矩时可进一步提升节省比例。此外，相较于同类方法，它加速了收敛速度并提升了下游任务性能。