Gradient methods are experiencing a growth in methodological and theoretical developments owing to the challenges of optimization problems arising in data science. Focusing on data science applications with expensive objective function evaluations yet inexpensive gradient function evaluations, gradient methods that never make objective function evaluations are either being rejuvenated or actively developed. However, as we show, such gradient methods are all susceptible to catastrophic divergence under realistic conditions for data science applications. In light of this, gradient methods which make use of objective function evaluations become more appealing, yet, as we show, can result in an exponential increase in objective evaluations between accepted iterates. As a result, existing gradient methods are poorly suited to the needs of optimization problems arising from data science. In this work, we address this gap by developing a generic methodology that economically uses objective function evaluations in a problem-driven manner to prevent catastrophic divergence and avoid an explosion in objective evaluations between accepted iterates. Our methodology allows for specific procedures that can make use of specific step size selection methodologies or search direction strategies, and we develop a novel step size selection methodology that is well-suited to data science applications. We show that a procedure resulting from our methodology is highly competitive with standard optimization methods on CUTEst test problems. We then show a procedure resulting from our methodology is highly favorable relative to standard optimization methods on optimization problems arising in our target data science applications. Thus, we provide a novel gradient methodology that is better suited to optimization problems arising in data science.

翻译：梯度方法因数据科学中的优化问题挑战而在方法论和理论发展上呈现增长态势。针对目标函数评估昂贵但梯度计算廉价的数据科学应用，当前正在复兴或积极探索无需目标函数评估的梯度方法。然而，我们研究表明，这类梯度方法在数据科学应用的实际条件下均易发生灾难性发散。鉴于此，利用目标函数评估的梯度方法更具吸引力，但我们的研究显示，这类方法可能导致接受迭代步之间的目标函数评估次数呈指数级增长。因此，现有梯度方法难以满足数据科学优化问题的需求。为解决这一缺陷，本文开发了一种通用方法论，以问题驱动方式经济地使用目标函数评估：既防止灾难性发散，又避免接受迭代步间目标函数评估的爆炸式增长。该方法论允许通过特定流程适配步长选择策略或搜索方向策略，并创新性地提出一种适用于数据科学应用的步长选择方法。实验表明，基于该方法的流程在CUTEst测试问题上与标准优化方法具有高度竞争力。针对目标数据科学应用中的优化问题，该方法所得流程相较于标准优化方法表现出显著优势。由此，我们提出了一种更适用于数据科学优化问题的新型梯度方法论。