Decision trees have found widespread application within the machine learning community due to their flexibility and interpretability. This paper is directed towards learning decision trees from data using a Bayesian approach, which is challenging due to the potentially enormous parameter space required to span all tree models. Several approaches have been proposed to combat this challenge, with one of the more successful being Markov chain Monte Carlo (MCMC) methods. The efficacy and efficiency of MCMC methods fundamentally rely on the quality of the so-called proposals, which is the focus of this paper. In particular, this paper investigates using a Hamiltonian Monte Carlo (HMC) approach to explore the posterior of Bayesian decision trees more efficiently by exploiting the geometry of the likelihood within a global update scheme. Two implementations of the novel algorithm are developed and compared to existing methods by testing against standard datasets in the machine learning and Bayesian decision tree literature. HMC-based methods are shown to perform favourably with respect to predictive test accuracy, acceptance rate, and tree complexity.

翻译：决策树因其灵活性和可解释性在机器学习领域得到了广泛应用。本文采用贝叶斯方法从数据中学习决策树，但由于覆盖所有树模型所需的参数空间可能极其庞大，这一任务颇具挑战性。已有多种方法应对这一难题，其中较为成功的一类是基于马尔可夫链蒙特卡洛（MCMC）方法。MCMC方法的有效性和效率从根本上取决于所谓提议分布的质量——这正是本文的研究焦点。具体而言，本文探究了利用哈密顿蒙特卡洛（HMC）方法，通过全局更新方案中似然的几何特性，更高效地探索贝叶斯决策树后验分布。我们开发了该新颖算法的两种实现方式，并通过机器学习与贝叶斯决策树文献中的标准数据集进行测试，与现有方法进行了对比。实验表明，基于HMC的方法在预测测试准确率、接受率和树复杂度方面均展现出优越性能。