Fractional Gradient Descent (FGD) offers a novel and promising way to accelerate optimization by incorporating fractional calculus into machine learning. Although FGD has shown encouraging initial results across various optimization tasks, it faces significant challenges with convergence behavior and hyperparameter selection. Moreover, the impact of its hyperparameters is not fully understood, and scheduling them is particularly difficult in non-convex settings such as neural network training. To address these issues, we propose a novel approach called Learning to Optimize Caputo Fractional Gradient Descent (L2O-CFGD), which meta-learns how to dynamically tune the hyperparameters of Caputo FGD (CFGD). Our method's meta-learned schedule outperforms CFGD with static hyperparameters found through an extensive search and, in some tasks, achieves performance comparable to a fully black-box meta-learned optimizer. L2O-CFGD can thus serve as a powerful tool for researchers to identify high-performing hyperparameters and gain insights on how to leverage the history-dependence of the fractional differential in optimization.

翻译：分数梯度下降（FGD）通过将分数阶微积分引入机器学习，为加速优化提供了一种新颖且前景广阔的方法。尽管FGD在各种优化任务中已展现出鼓舞人心的初步成果，但其在收敛行为和超参数选择方面仍面临显著挑战。此外，其超参数的影响尚未被完全理解，在神经网络训练等非凸场景中调度这些参数尤为困难。为解决这些问题，我们提出了一种名为“学习优化卡普托分数梯度下降”（L2O-CFGD）的新方法，该方法通过元学习动态调整卡普托FGD（CFGD）的超参数。我们方法中通过元学习得到的调度策略，其性能优于经过广泛搜索确定的静态超参数CFGD，并且在某些任务中达到了与完全黑盒元学习优化器相当的性能。因此，L2O-CFGD可作为一个强大工具，帮助研究者识别高性能超参数，并深入理解如何利用分数阶微分的历史依赖性来优化性能。