Gradient boosting machines (GBMs) based on decision trees consistently demonstrate state-of-the-art results on regression and classification tasks with tabular data, often outperforming deep neural networks. However, these models do not provide well-calibrated predictive uncertainties, which prevents their use for decision making in high-risk applications. The Bayesian treatment is known to improve predictive uncertainty calibration, but previously proposed Bayesian GBM methods are either computationally expensive, or resort to crude approximations. Variational inference is often used to implement Bayesian neural networks, but is difficult to apply to GBMs, because the decision trees used as weak learners are non-differentiable. In this paper, we propose to implement Bayesian GBMs using variational inference with soft decision trees, a fully differentiable alternative to standard decision trees introduced by Irsoy et al. Our experiments demonstrate that variational soft trees and variational soft GBMs provide useful uncertainty estimates, while retaining good predictive performance. The proposed models show higher test likelihoods when compared to the state-of-the-art Bayesian GBMs in 7/10 tabular regression datasets and improved out-of-distribution detection in 5/10 datasets.

翻译：基于决策树的梯度提升机（GBMs）在表格数据的回归和分类任务中持续展现出最先进的性能，通常优于深度神经网络。然而，这些模型无法提供良好校准的预测不确定性，这阻碍了它们在高风险应用决策中的使用。贝叶斯方法已知能改善预测不确定性校准，但先前提出的贝叶斯GBM方法要么计算成本高昂，要么采用粗略的近似方法。变分推断常用于实现贝叶斯神经网络，但难以应用于GBM，因为用作弱学习器的决策树是不可微的。本文提出使用软决策树——由Irsoy等人引入的标准决策树的完全可微替代方案——通过变分推断实现贝叶斯GBM。实验表明，变分软树和变分软GBM在保持良好预测性能的同时，提供了有用的不确定性估计。与最先进的贝叶斯GBM相比，所提出的模型在7/10的表格回归数据集上显示出更高的测试似然，并在5/10的数据集上改进了分布外检测。