We present \emph{TabRet}, a pre-trainable Transformer-based model for tabular data. TabRet is designed to work on a downstream task that contains columns not seen in pre-training. Unlike other methods, TabRet has an extra learning step before fine-tuning called \emph{retokenizing}, which calibrates feature embeddings based on the masked autoencoding loss. In experiments, we pre-trained TabRet with a large collection of public health surveys and fine-tuned it on classification tasks in healthcare, and TabRet achieved the best AUC performance on four datasets. In addition, an ablation study shows retokenizing and random shuffle augmentation of columns during pre-training contributed to performance gains. The code is available at https://github.com/pfnet-research/tabret .

翻译：我们提出\emph{TabRet}，一种可预训练的基于Transformer的表格数据模型。TabRet旨在处理包含预训练中未见列的下游任务。与其他方法不同，TabRet在微调前增加一个名为\emph{重分词}的额外学习步骤，该步骤基于掩码自编码损失校准特征嵌入。实验中，我们使用大规模公共卫生调查数据预训练TabRet，并在医疗健康分类任务上对其进行微调，TabRet在四个数据集上取得了最佳AUC性能。此外，消融研究表明，重分词与预训练期间对列进行随机洗牌增强有助于性能提升。代码详见https://github.com/pfnet-research/tabret。