Network optimization is a crucial step in the field of deep learning, as it directly affects the performance of models in various domains such as computer vision. Despite the numerous optimizers that have been developed over the years, the current methods are still limited in their ability to accurately and quickly identify gradient trends, which can lead to sub-optimal network performance. In this paper, we propose a novel deep optimizer called Fast-Adaptive Moment Estimation (FAME), which for the first time estimates gradient moments using a Triple Exponential Moving Average (TEMA). Incorporating TEMA into the optimization process provides richer and more accurate information on data changes and trends, as compared to the standard Exponential Moving Average used in essentially all current leading adaptive optimization methods. Our proposed FAME optimizer has been extensively validated through a wide range of benchmarks, including CIFAR-10, CIFAR-100, PASCAL-VOC, MS-COCO, and Cityscapes, using 14 different learning architectures, six optimizers, and various vision tasks, including detection, classification and semantic understanding. The results demonstrate that our FAME optimizer outperforms other leading optimizers in terms of both robustness and accuracy.

翻译：网络优化是深度学习领域的关键步骤，直接影响模型在计算机视觉等多个领域的性能。尽管多年来已开发出众多优化器，但现有方法在准确快速识别梯度趋势方面仍存在局限性，这可能导致网络性能欠优。本文提出一种新型深度优化器——快速自适应矩估计（FAME），首次采用三重指数移动平均（TEMA）估计梯度矩。相较于当前主流自适应优化方法中普遍使用的标准指数移动平均，将TEMA引入优化过程可提供更丰富、更准确的数据变化与趋势信息。我们提出的FAME优化器已通过CIFAR-10、CIFAR-100、PASCAL-VOC、MS-COCO和Cityscapes等广泛基准测试，使用14种不同学习架构、六种优化器以及检测、分类和语义理解等多种视觉任务进行了全面验证。结果表明，我们的FAME优化器在鲁棒性和准确性方面均优于其他主流优化器。