The optimistic gradient method has seen increasing popularity for solving convex-concave saddle point problems. To analyze its iteration complexity, a recent work [arXiv:1906.01115] proposed an interesting perspective that interprets this method as an approximation to the proximal point method. In this paper, we follow this approach and distill the underlying idea of optimism to propose a generalized optimistic method, which includes the optimistic gradient method as a special case. Our general framework can handle constrained saddle point problems with composite objective functions and can work with arbitrary norms using Bregman distances. Moreover, we develop a backtracking line search scheme to select the step sizes without knowledge of the smoothness coefficients. We instantiate our method with first-, second- and higher-order oracles and give best-known global iteration complexity bounds. For our first-order method, we show that the averaged iterates converge at a rate of $O(1/N)$ when the objective function is convex-concave, and it achieves linear convergence when the objective is strongly-convex-strongly-concave. For our second- and higher-order methods, under the additional assumption that the distance-generating function has Lipschitz gradient, we prove a complexity bound of $O(1/\epsilon^\frac{2}{p+1})$ in the convex-concave setting and a complexity bound of $O((L_pD^\frac{p-1}{2}/\mu)^\frac{2}{p+1}+\log\log\frac{1}{\epsilon})$ in the strongly-convex-strongly-concave setting, where $L_p$ ($p\geq 2$) is the Lipschitz constant of the $p$-th-order derivative, $\mu$ is the strong convexity parameter, and $D$ is the initial Bregman distance to the saddle point. Moreover, our line search scheme provably only requires a constant number of calls to a subproblem solver per iteration on average, making our first- and second-order methods particularly amenable to implementation.

翻译：乐观梯度法在求解凸-凹鞍点问题中日益流行。为分析其迭代复杂度，近期工作[arXiv:1906.01115]提出了一个有趣视角，将该方法解释为近端点法的近似。本文沿袭这一思路，提炼乐观思想的核心精髓，提出包含乐观梯度法作为特例的广义乐观方法。该通用框架能处理含复合目标函数的约束鞍点问题，并可通过Bregman距离兼容任意范数。此外，我们开发了一种无需光滑系数先验知识的回溯线搜索步长选择方案。我们分别用一阶、二阶及高阶预言机实例化该方法，并给出了当前最优的全局迭代复杂度界。对于一阶方法，我们证明当目标函数为凸-凹时，平均迭代点的收敛率为$O(1/N)$；当目标函数为强凸-强凹时，可实现线性收敛。对于二阶及高阶方法，在距离生成函数具有Lipschitz梯度的附加假设下，我们证明凸-凹情形下的复杂度界为$O(1/\epsilon^\frac{2}{p+1})$，强凸-强凹情形下的复杂度界为$O((L_pD^\frac{p-1}{2}/\mu)^\frac{2}{p+1}+\log\log\frac{1}{\epsilon})$，其中$L_p$（$p\geq 2$）为$p$阶导数的Lipschitz常数，$\mu$为强凸参数，$D$为初始Bregman距离至鞍点的距离。此外，我们的线搜索方案在平均每次迭代中仅需常数次调用子问题求解器，使一阶和二阶方法特别适合实际实现。