Tabular data poses unique challenges for deep learning due to its heterogeneous feature types, lack of spatial structure, and often limited sample sizes. We propose TabNSA, a novel deep learning framework that integrates Native Sparse Attention (NSA) with a TabMixer backbone to efficiently model tabular data. TabNSA tackles computational and representational challenges by dynamically focusing on relevant feature subsets per instance. The NSA module employs a hierarchical sparse attention mechanism, including token compression, selective preservation, and localized sliding windows, to significantly reduce the quadratic complexity of standard attention operations while addressing feature heterogeneity. Complementing this, the TabMixer backbone captures complex, non-linear dependencies through parallel multilayer perceptron (MLP) branches with independent parameters. These modules are synergistically combined via element-wise summation and mean pooling, enabling TabNSA to model both global context and fine-grained interactions. Extensive experiments across supervised and transfer learning settings show that TabNSA consistently outperforms state-of-the-art deep learning models. Furthermore, by augmenting TabNSA with a fine-tuned large language model (LLM), we enable it to effectively address Few-Shot Learning challenges through language-guided generalization on diverse tabular benchmarks. Code available on: https://github.com/aseslamian/TabNSA

翻译：表格数据因其特征类型异构、缺乏空间结构且样本规模通常有限，对深度学习提出了独特挑战。本文提出TabNSA——一种集成原生稀疏注意力与TabMixer主干网络的新型深度学习框架，用于高效建模表格数据。TabNSA通过动态聚焦每个实例的相关特征子集，应对计算与表征层面的双重挑战。NSA模块采用包含令牌压缩、选择性保留和局部滑动窗口的分层稀疏注意力机制，在应对特征异构性的同时，显著降低了标准注意力操作的二次复杂度。与之互补的TabMixer主干通过具有独立参数的并行多层感知机分支捕获复杂的非线性依赖关系。这些模块通过逐元素求和与均值池化进行协同整合，使TabNSA能够同时建模全局上下文与细粒度交互。在监督学习与迁移学习场景下的大规模实验表明，TabNSA始终优于当前最先进的深度学习模型。此外，通过为TabNSA集成精调的大型语言模型，我们使其能够通过语言引导的泛化机制，在多类表格基准数据集上有效应对少样本学习挑战。代码发布于：https://github.com/aseslamian/TabNSA