As the complexity of learning tasks surges, modern machine learning encounters a new constrained learning paradigm characterized by more intricate and data-driven function constraints. Prominent applications include Neyman-Pearson classification (NPC) and fairness classification, which entail specific risk constraints that render standard projection-based training algorithms unsuitable. Gradient boosting machines (GBMs) are among the most popular algorithms for supervised learning; however, they are generally limited to unconstrained settings. In this paper, we adapt the GBM for constrained learning tasks within the framework of Bregman proximal algorithms. We introduce a new Bregman primal-dual method with a global optimality guarantee when the learning objective and constraint functions are convex. In cases of nonconvex functions, we demonstrate how our algorithm remains effective under a Bregman proximal point framework. Distinct from existing constrained learning algorithms, ours possess a unique advantage in their ability to seamlessly integrate with publicly available GBM implementations such as XGBoost (Chen and Guestrin, 2016) and LightGBM (Ke et al., 2017), exclusively relying on their public interfaces. We provide substantial experimental evidence to showcase the effectiveness of the Bregman algorithm framework. While our primary focus is on NPC and fairness ML, our framework holds significant potential for a broader range of constrained learning applications. The source code is currently freely available at https://github.com/zhenweilin/ConstrainedGBM}{https://github.com/zhenweilin/ConstrainedGBM.

翻译：随着学习任务复杂性的急剧增加，现代机器学习面临一种新的约束学习范式，其特征是更加复杂且数据驱动的函数约束。典型应用包括Neyman-Pearson分类（NPC）和公平性分类，这些应用涉及特定的风险约束，使得基于投影的标准训练算法不再适用。梯度提升机（GBM）是监督学习中最流行的算法之一，然而它们通常局限于无约束场景。在本文中，我们基于Bregman邻近算法框架，将GBM适配于约束学习任务。我们提出了一种新的Bregman原始-对偶方法，在学习目标和约束函数为凸函数时，该方法具有全局最优性保证。对于非凸函数情形，我们展示了在Bregman邻近点框架下，算法如何保持有效。与现有约束学习算法不同，我们的算法具有独特优势：能够无缝集成公开可用的GBM实现，如XGBoost（Chen和Guestrin, 2016）和LightGBM（Ke等，2017），且仅依赖其公共接口。我们提供大量实验证据，展示了Bregman算法框架的有效性。虽然本文主要关注NPC和公平性机器学习，但我们的框架对更广泛的约束学习应用具有重要潜力。源代码目前可在https://github.com/zhenweilin/ConstrainedGBM}{https://github.com/zhenweilin/ConstrainedGBM免费获取。