In this paper we discuss the Anderson's type acceleration method for numerical optimizations. Most mathematical modeling problems can be formulated as constrained optimization. The necessary optimality condition is written as a fixed point problem in a Banach space. Anderson's acceleration method improves the convergence of the standard fixed point iteration by minimizing the total sum of residuals and updating solutions through an optimal linear combination of a sequence of iterates. Thus, it is a form of iterative method of retard which uses the history of solutions as a reduced order element method (ROM). The weights are determined optimally by the least squares problem based on the total residual. We analyze Anderson's method and the reduced order method (ROM) for nonlinear least squares problems of minimizing |F(x)| squared. That is, the solution is approximated by a linear combination of sequentially generated solutions, and then we minimize the equation error on the linear manifold spanned by the iterates. We use the reduced order Gauss Newton method to solve the least squares problem for |F(x)| squared on the linear solution manifold. For the linear equation case it is similar to Anderson's method. Anderson's method approximates the solution to the nonlinear ROM. We consider variable step size gradient and quasi Newton methods and the variable fixed point iteration to generate the solution basis, then combine these with ROM acceleration. It converges very rapidly if the condition number of matrix A is not large. The variable iterate with ROM acceleration is nearly optimal, and ROM also regularizes and stabilizes the convergence. We also consider a randomly overlapped Kaczmarz type method and develop an acceleration approach for large scale ill posed linear systems. Finally, we analyze the convergence of the ROM to the operator equation.

翻译：本文探讨了数值优化中的Anderson型加速方法。大多数数学建模问题均可表述为约束优化问题，其必要最优性条件可写为巴拿赫空间中的不动点问题。Anderson加速方法通过最小化残差总和，并利用迭代序列的最优线性组合更新解，从而改进标准不动点迭代的收敛性。因此，这是一种采用历史解作为降阶元方法（ROM）的延迟迭代法。权重基于总残差通过最小二乘问题最优确定。我们分析了针对最小化|F(x)|平方的非线性最小二乘问题的Anderson方法及降阶方法（ROM）。该方法通过顺序生成解的线性组合逼近解，并在迭代张成的线性流形上最小化方程误差。我们采用降阶高斯-牛顿法求解线性解流形上|F(x)|平方的最小二乘问题。对于线性方程情形，该方法与Anderson方法具有相似性。Anderson方法可视为非线性ROM的近似解。我们考虑采用变步长梯度法、拟牛顿法及可变不动点迭代生成解基，再与ROM加速相结合。当矩阵A的条件数较小时，该方法收敛速度极快。结合ROM加速的变步长迭代近乎最优，且ROM能正则化并稳定收敛过程。我们还研究了随机重叠Kaczmarz型方法，并针对大规模不适定线性系统提出了加速策略。最后，我们分析了ROM对算子方程的收敛性。