Decision trees remain one of the most popular machine learning models today, largely due to their out-of-the-box performance and interpretability. In this work, we present a Bayesian approach to decision tree induction via maximum a posteriori inference of a posterior distribution over trees. We first demonstrate a connection between maximum a posteriori inference of decision trees and AND/OR search. Using this connection, we propose an AND/OR search algorithm, dubbed MAPTree, which is able to recover the maximum a posteriori tree. Lastly, we demonstrate the empirical performance of the maximum a posteriori tree both on synthetic data and in real world settings. On 16 real world datasets, MAPTree either outperforms baselines or demonstrates comparable performance but with much smaller trees. On a synthetic dataset, MAPTree also demonstrates greater robustness to noise and better generalization than existing approaches. Finally, MAPTree recovers the maxiumum a posteriori tree faster than existing sampling approaches and, in contrast with those algorithms, is able to provide a certificate of optimality. The code for our experiments is available at https://github.com/ThrunGroup/maptree.

翻译：决策树至今仍是机器学习领域最受欢迎的模型之一，主要得益于其即开即用的性能与可解释性。本文提出一种基于贝叶斯方法的决策树归纳框架，通过树的后验分布的最大后验推断实现模型构建。我们首先证明了决策树的最大后验推断与AND/OR搜索之间的内在联系，进而基于此关联提出名为MAPTree的AND/OR搜索算法，该算法能够恢复最大后验树。最后，我们在合成数据与真实场景中验证了最大后验树的实证表现：在16个真实数据集上，MAPTree或优于基线模型，或以更小的树规模实现可比性能；在合成数据集上，MAPTree相比现有方法展现出更强的抗噪性能与泛化能力。此外，与现有采样方法相比，MAPTree能更快恢复最大后验树，并能提供最优性证明。实验代码已发布于https://github.com/ThrunGroup/maptree。