Black-box algorithms are designed to optimize functions without relying on their underlying analytical structure or gradient information, making them essential when gradients are inaccessible or difficult to compute. Traditional methods for solving black-box optimization (BBO) problems predominantly rely on non-parametric models and struggle to scale to large input spaces. Conversely, parametric methods that model the function with neural estimators and obtain gradient signals via backpropagation may suffer from significant gradient errors. A recent alternative, Explicit Gradient Learning (EGL), which directly learns the gradient using a first-order Taylor approximation, has demonstrated superior performance over both parametric and non-parametric methods. In this work, we propose two novel gradient learning variants to address the robustness challenges posed by high-dimensional, complex, and highly non-linear problems. Optimistic Gradient Learning (OGL) introduces a bias toward lower regions in the function landscape, while Higher-order Gradient Learning (HGL) incorporates second-order Taylor corrections to improve gradient accuracy. We combine these approaches into the unified OHGL algorithm, achieving state-of-the-art (SOTA) performance on the synthetic COCO suite. Additionally, we demonstrate OHGLs applicability to high-dimensional real-world machine learning (ML) tasks such as adversarial training and code generation. Our results highlight OHGLs ability to generate stronger candidates, offering a valuable tool for ML researchers and practitioners tackling high-dimensional, non-linear optimization challenges

翻译：黑盒算法旨在优化函数时无需依赖其底层解析结构或梯度信息，这使得它们在梯度无法获取或难以计算时至关重要。解决黑盒优化问题的传统方法主要依赖于非参数模型，难以扩展至大规模输入空间。相反，使用神经估计器建模函数并通过反向传播获取梯度信号的参数化方法可能受到显著梯度误差的影响。近期提出的显式梯度学习方法通过一阶泰勒近似直接学习梯度，已展现出优于参数化和非参数化方法的性能。在本研究中，我们提出了两种新颖的梯度学习变体，以应对高维、复杂和高度非线性问题带来的鲁棒性挑战。乐观梯度学习方法引入了对函数景观中较低区域的偏向，而高阶梯度学习方法则通过二阶泰勒修正来提高梯度精度。我们将这些方法整合为统一的OHGL算法，在合成COCO测试套件上实现了最先进的性能。此外，我们展示了OHGL在高维现实世界机器学习任务中的适用性，例如对抗性训练和代码生成。我们的研究结果突显了OHGL生成更强候选解的能力，为应对高维非线性优化挑战的机器学习研究者和实践者提供了有价值的工具。