We introduce two new stochastic conjugate frameworks for a class of nonconvex and possibly also nonsmooth optimization problems. These frameworks are built upon Stochastic Recursive Gradient Algorithm (SARAH) and we thus refer to them as Acc-Prox-CG-SARAH and Acc-Prox-CG-SARAH-RS, respectively. They are efficiently accelerated, easy to implement, tune free and can be smoothly extended and modified. We devise a deterministic restart scheme for stochastic optimization and apply it in our second stochastic conjugate framework, which serves the key difference between the two approaches. In addition, we apply the ProbAbilistic Gradient Estimator (PAGE) and further develop a practical variant, denoted as Acc-Prox-CG-SARAH-ST, in order to reduce potential computational overhead. We provide comprehensive and rigorous convergence analysis for all three approaches and establish linear convergence rates for unconstrained minimization problem with nonconvex and nonsmooth objective functions. Experiments have demonstrated that Acc-Prox-CG-SARAH and Acc-Prox-CG-SARAH-RS both outperform state-of-art methods consistently and Acc-Prox-CG-SARAH-ST can as well achieve comparable convergence speed. In terms of theory and experiments, we verify the strong computational efficiency of the deterministic restart scheme in stochastic optimization methods.

翻译：我们针对一类非凸且可能非光滑的优化问题提出了两种新的随机共轭框架。这两种框架基于随机递归梯度算法（SARAH）构建，因此分别命名为Acc-Prox-CG-SARAH和Acc-Prox-CG-SARAH-RS。它们具有高效加速、易于实现、无需调参的优势，并能平滑扩展与修改。我们为随机优化设计了一种确定性重启方案，并将其应用于第二个随机共轭框架中，这是两种方法的主要区别。此外，我们引入概率梯度估计器（PAGE）进一步开发了一个实用变体Acc-Prox-CG-SARAH-ST，以降低潜在计算开销。我们为所有三种方法提供了全面且严格的收敛性分析，并针对非凸非光滑目标函数的无约束最小化问题建立了线性收敛速率。实验表明，Acc-Prox-CG-SARAH和Acc-Prox-CG-SARAH-RH均持续优于现有最优方法，而Acc-Prox-CG-SARAH-ST也能达到相当的收敛速度。在理论与实验层面，我们验证了随机优化方法中确定性重启方案的高计算效率。