Gradient Boosting Machines (GBMs) have demonstrated remarkable success in solving diverse problems by utilizing Taylor expansions in functional space. However, achieving a balance between performance and generality has posed a challenge for GBMs. In particular, gradient descent-based GBMs employ the first-order Taylor expansion to ensure applicability to all loss functions, while Newton's method-based GBMs use positive Hessian information to achieve superior performance at the expense of generality. To address this issue, this study proposes a new generic Gradient Boosting Machine called Trust-region Boosting (TRBoost). In each iteration, TRBoost uses a constrained quadratic model to approximate the objective and applies the Trust-region algorithm to solve it and obtain a new learner. Unlike Newton's method-based GBMs, TRBoost does not require the Hessian to be positive definite, thereby allowing it to be applied to arbitrary loss functions while still maintaining competitive performance similar to second-order algorithms. The convergence analysis and numerical experiments conducted in this study confirm that TRBoost is as general as first-order GBMs and yields competitive results compared to second-order GBMs. Overall, TRBoost is a promising approach that balances performance and generality, making it a valuable addition to the toolkit of machine learning practitioners.

翻译：梯度提升机（GBMs）通过在函数空间中利用泰勒展开，在解决各类问题中展现出卓越成效。然而，在性能与通用性之间寻求平衡始终是GBMs面临的挑战。具体而言，基于梯度下降的GBM采用一阶泰勒展开以确保适用于所有损失函数，而基于牛顿法的GBM则通过使用正定Hessian矩阵信息来获得更优性能，但牺牲了通用性。为解决此问题，本研究提出一种名为信赖域提升（TRBoost）的新型通用梯度提升机。在每次迭代中，TRBoost使用带约束的二次模型近似目标函数，并应用信赖域算法对其进行求解以获取新的学习器。与基于牛顿法的GBM不同，TRBoost无需Hessian矩阵正定，因而可应用于任意损失函数，同时保持与二阶算法相媲美的竞争性能。本研究的收敛性分析与数值实验证实，TRBoost具备与一阶GBM同等的通用性，且能获得与二阶GBM相近的竞争结果。总体而言，TRBoost是一种兼顾性能与通用性的有前途方法，为机器学习实践者的工具箱增添了宝贵方案。