We present a new algorithm for automatically bounding the Taylor remainder series. In the special case of a scalar function $f: \mathbb{R} \to \mathbb{R}$, our algorithm takes as input a reference point $x_0$, trust region $[a, b]$, and integer $k \ge 1$, and returns an interval $I$ such that $f(x) - \sum_{i=0}^{k-1} \frac {1} {i!} f^{(i)}(x_0) (x - x_0)^i \in I (x - x_0)^k$ for all $x \in [a, b]$. As in automatic differentiation, the function $f$ is provided to the algorithm in symbolic form, and must be composed of known atomic functions. At a high level, our algorithm has two steps. First, for a variety of commonly-used elementary functions (e.g., $\exp$, $\log$), we derive sharp polynomial upper and lower bounds on the Taylor remainder series. We then recursively combine the bounds for the elementary functions using an interval arithmetic variant of Taylor-mode automatic differentiation. Our algorithm can make efficient use of machine learning hardware accelerators, and we provide an open source implementation in JAX. We then turn our attention to applications. Most notably, we use our new machinery to create the first universal majorization-minimization optimization algorithms: algorithms that iteratively minimize an arbitrary loss using a majorizer that is derived automatically, rather than by hand. Applied to machine learning, this leads to architecture-specific optimizers for training deep networks that converge from any starting point, without hyperparameter tuning. Our experiments show that for some optimization problems, these hyperparameter-free optimizers outperform tuned versions of gradient descent, Adam, and AdaGrad. We also show that our automatically-derived bounds can be used for verified global optimization and numerical integration, and to prove sharper versions of Jensen's inequality.

翻译：我们提出了一种新算法，用于自动界定泰勒余项级数。在标量函数$f: \mathbb{R} \to \mathbb{R}$的特殊情形下，算法输入参考点$x_0$、信赖域$[a, b]$以及整数$k \ge 1$，并返回一个区间$I$，使得对所有$x \in [a, b]$，有$f(x) - \sum_{i=0}^{k-1} \frac {1} {i!} f^{(i)}(x_0) (x - x_0)^i \in I (x - x_0)^k$。与自动微分类似，函数$f$以符号形式提供给算法，且必须由已知的原子函数复合而成。从高层面看，算法包含两个步骤。首先，针对多种常用初等函数（如$\exp$、$\log$），我们推导出泰勒余项级数的精确多项式上界与下界。随后，我们利用泰勒模式自动微分的区间算术变体，递归组合这些初等函数的界。本算法可高效利用机器学习硬件加速器，并提供基于JAX的开源实现。接下来，我们将其应用于实际问题。最值得注意的是，我们利用这一新机制首次提出通用化最小化-最大化优化算法：该算法通过自动推导而非手动构造的优函数，迭代地最小化任意损失函数。应用于机器学习时，该算法能生成针对特定架构的优化器，用于训练深度网络，使其从任意初始点收敛且无需超参数调整。实验表明，针对某些优化问题，这些免超参数优化器的性能优于调参后的梯度下降、Adam和AdaGrad。我们还证明，自动推导的界可用于验证全局优化、数值积分，以及证明更紧的Jensen不等式形式。