The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results on various datasets. However, the proposed models are usually not properly compared to each other and existing works often use different benchmarks and experiment protocols. As a result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems. In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is a ResNet-like architecture which turns out to be a strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures on a diverse set of tasks under the same training and tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution.

翻译：现有关于表格数据深度学习的文献提出了多种新颖架构，并在不同数据集上报告了有竞争力的结果。然而，这些模型通常未得到合理对比，且现有工作常采用不同的基准测试集和实验协议。因此，研究者和实践者均难以明确何种模型表现最优。此外，该领域仍缺乏有效的基线模型——即能在不同问题上保持竞争力的易用型模型。本研究系统梳理了表格数据深度学习架构的主要类别，并通过识别两种简单而强大的深度架构提升了表格深度学习基线的水平。第一种是基于残差网络的架构，事实证明它常被先前研究忽视却具备强基线性能。第二种模型是我们在Transformer架构基础上针对表格数据的简易改进，在多数任务上优于其他解决方案。我们在统一训练与调参协议下，将这两种模型与多种现有架构在多样化任务集上进行了对比。同时将最优深度学习模型与梯度提升决策树进行横向比较，结论是当前尚不存在通用的最优方案。