This study surveys the historical development of regularization, tracing its evolution from stepwise regression in the 1960s to recent advancements in formal error control, structured penalties for non-independent features, Bayesian methods, and l0-based regularization (among other techniques). We empirically evaluate the performance of four canonical frameworks -- Ridge, Lasso, ElasticNet, and Post-Lasso OLS -- across 134,400 simulations spanning a 7-dimensional manifold grounded in eight production-grade machine learning models. Our findings demonstrate that for prediction accuracy when the sample-to-feature ratio is sufficient (n/p >= 78), Ridge, Lasso, and ElasticNet are nearly interchangeable. However, we find that Lasso recall is highly fragile under multicollinearity; at high condition numbers (kappa) and low SNR, Lasso recall collapses to 0.18 while ElasticNet maintains 0.93. Consequently, we advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes. The analysis concludes with an objective-driven decision guide to assist machine learning engineers in selecting the optimal scikit-learn-supported framework based on observable feature space attributes.

翻译：本研究梳理了正则化方法的历史演进，追溯其从20世纪60年代的逐步回归到近期在形式化误差控制、针对非独立特征的结构化惩罚、贝叶斯方法以及基于l0的正则化（涵盖多种技术）的进展。我们通过跨越134,400次模拟（基于七维流形空间，该空间以八种生产级机器学习模型为基础），实证评估了四种经典框架——岭回归、套索回归、弹性网络以及后套索普通最小二乘法。研究结果表明：在样本量充足（样本特征比n/p >= 78）且以预测精度为目标的场景下，岭回归、套索回归与弹性网络的性能几乎可互换。然而，我们发现套索回归的召回率在多重共线性条件下高度脆弱：当条件数较高且信噪比较低时，其召回率降至0.18，而弹性网络仍保持0.93。因此，我们建议从业者避免在条件数高且样本量小的场景中使用套索回归或后套索普通最小二乘法。本分析最终提出了一套目标导向的决策指南，以帮助机器学习工程师根据可观测特征空间属性，选择最优的scikit-learn框架。