Despite their success with unstructured data, deep neural networks are not yet a panacea for structured tabular data. In the tabular domain, their efficiency crucially relies on various forms of regularization to prevent overfitting and provide strong generalization performance. Existing regularization techniques include broad modelling decisions such as choice of architecture, loss functions, and optimization methods. In this work, we introduce Tabular Neural Gradient Orthogonalization and Specialization (TANGOS), a novel framework for regularization in the tabular setting built on latent unit attributions. The gradient attribution of an activation with respect to a given input feature suggests how the neuron attends to that feature, and is often employed to interpret the predictions of deep networks. In TANGOS, we take a different approach and incorporate neuron attributions directly into training to encourage orthogonalization and specialization of latent attributions in a fully-connected network. Our regularizer encourages neurons to focus on sparse, non-overlapping input features and results in a set of diverse and specialized latent units. In the tabular domain, we demonstrate that our approach can lead to improved out-of-sample generalization performance, outperforming other popular regularization methods. We provide insight into why our regularizer is effective and demonstrate that TANGOS can be applied jointly with existing methods to achieve even greater generalization performance.

翻译：尽管深度神经网络在非结构化数据上取得了成功，但其尚未成为结构化表格数据的通用解决方案。在表格领域，深度神经网络的效能关键依赖于各类正则化技术以防止过拟合并实现强泛化性能。现有正则化技术涵盖广泛的设计决策，如架构选择、损失函数和优化方法。本文提出表格神经梯度正交化与特化（TANGOS），一种基于潜在单元归因的表格场景正则化新框架。激活值相对于给定输入特征的梯度归因反映了神经元对该特征的关注程度，常被用于解释深度网络的预测。在TANGOS中，我们采用不同方法，直接将神经元归因纳入训练过程，以促进全连接网络中潜在归因的正交化与特化。该正则化器鼓励神经元聚焦于稀疏、非重叠的输入特征，从而产生一组多样化且特化的潜在单元。我们在表格领域证明，该方法能提升样本外泛化性能，优于其他主流正则化技术。我们深入解析了该正则化器有效性的原因，并证明TANGOS可与现有方法联合应用，实现更强的泛化性能。