The analytic characterization of the high-dimensional behavior of optimization for Generalized Linear Models (GLMs) with Gaussian data has been a central focus in statistics and probability in recent years. While convex cases, such as the LASSO, ridge regression, and logistic regression, have been extensively studied using a variety of techniques, the non-convex case remains far less understood despite its significance. A non-rigorous statistical physics framework has provided remarkable predictions for the behavior of high-dimensional optimization problems, but rigorously establishing their validity for non-convex problems has remained a fundamental challenge. In this work, we address this challenge by developing a systematic framework that rigorously proves replica-symmetric formulas for non-convex GLMs and precisely determines the conditions under which these formulas are valid. Remarkably, the rigorous replica-symmetric predictions align exactly with the conjectures made by physicists, and the so-called replicon condition. The originality of our approach lies in connecting two powerful theoretical tools: the Gaussian Min-Max Theorem, which we use to provide precise lower bounds, and Approximate Message Passing (AMP), which is shown to achieve these bounds algorithmically. We demonstrate the utility of this framework through significant applications: (i) by proving the optimality of the Tukey loss over the more commonly used Huber loss under a $\varepsilon$ contaminated data model, (ii) establishing the optimality of negative regularization in high-dimensional non-convex regression and (iii) characterizing the performance limits of linearized AMP algorithms. By rigorously validating statistical physics predictions in non-convex settings, we aim to open new pathways for analyzing increasingly complex optimization landscapes beyond the convex regime.

翻译：近年来，高斯数据下广义线性模型高维优化行为的解析刻画已成为统计学和概率论的核心研究焦点。尽管凸情形（如LASSO、岭回归和逻辑回归）已通过多种技术得到广泛研究，但非凸情形尽管具有重要意义，其理解仍远不充分。非严格的统计物理框架为高维优化问题的行为提供了卓越的预测，但严格证明这些预测在非凸问题中的有效性仍是一个根本性挑战。本文通过构建一个系统性框架应对这一挑战，该框架严格证明了非凸GLMs的副本对称公式，并精确确定了这些公式有效的条件。值得注意的是，严格的副本对称预测与物理学家的猜想及所谓的副本条件完全吻合。我们方法的原创性在于连接了两个强大的理论工具：用于提供精确下界的高斯极小极大定理，以及被证明能以算法方式达到这些下界的近似消息传递算法。我们通过以下重要应用展示了该框架的实用性：(i) 在$\varepsilon$污染数据模型下证明Tukey损失比更常用的Huber损失具有最优性，(ii) 确立高维非凸回归中负正则化的最优性，以及(iii) 刻画线性化AMP算法的性能极限。通过在非凸场景中严格验证统计物理预测，我们旨在为分析超越凸区域的日益复杂的优化景观开辟新途径。