Gradient Boosting Machines (GBMs) have demonstrated remarkable success in solving diverse problems by utilizing Taylor expansions in functional space. However, achieving a balance between performance and generality has posed a challenge for GBMs. In particular, gradient descent-based GBMs employ the first-order Taylor expansion to ensure applicability to all loss functions, while Newton's method-based GBMs use positive Hessian information to achieve superior performance at the expense of generality. To address this issue, this study proposes a new generic Gradient Boosting Machine called Trust-region Boosting (TRBoost). In each iteration, TRBoost uses a constrained quadratic model to approximate the objective and applies the Trust-region algorithm to solve it and obtain a new learner. Unlike Newton's method-based GBMs, TRBoost does not require the Hessian to be positive definite, thereby allowing it to be applied to arbitrary loss functions while still maintaining competitive performance similar to second-order algorithms. The convergence analysis and numerical experiments conducted in this study confirm that TRBoost is as general as first-order GBMs and yields competitive results compared to second-order GBMs. Overall, TRBoost is a promising approach that balances performance and generality, making it a valuable addition to the toolkit of machine learning practitioners.

翻译：梯度提升机通过利用函数空间中的泰勒展开，在解决各类问题中展现出显著成功。然而，在性能与通用性之间取得平衡一直是梯度提升机面临的挑战。具体而言，基于梯度下降的梯度提升机采用一阶泰勒展开以确保适用于所有损失函数，而基于牛顿法的梯度提升机则使用正定的海森矩阵信息以获得更优性能，但这牺牲了通用性。为解决此问题，本研究提出一种新的通用梯度提升机——信赖域提升（TRBoost）。在每次迭代中，TRBoost使用带约束的二次模型逼近目标函数，并运用信赖域算法求解该模型以获得新的学习器。与基于牛顿法的梯度提升机不同，TRBoost不要求海森矩阵正定，因而可应用于任意损失函数，同时仍保持与二阶算法相当的竞争性能。本研究的收敛性分析及数值实验证实，TRBoost具有与一阶梯度提升机相同的通用性，且能达到与二阶梯度提升机竞争的结果。总体而言，TRBoost是一种在性能与通用性间取得平衡的有效方法，为机器学习实践者提供了有价值的工具。