For machine learning with tabular data, Table Transformer (TabTransformer) is a state-of-the-art neural network model, while Differential Privacy (DP) is an essential component to ensure data privacy. In this paper, we explore the benefits of combining these two aspects together in the scenario of transfer learning -- differentially private pre-training and fine-tuning of TabTransformers with a variety of parameter-efficient fine-tuning (PEFT) methods, including Adapter, LoRA, and Prompt Tuning. Our extensive experiments on the ACSIncome dataset show that these PEFT methods outperform traditional approaches in terms of the accuracy of the downstream task and the number of trainable parameters, thus achieving an improved trade-off among parameter efficiency, privacy, and accuracy. Our code is available at github.com/IBM/DP-TabTransformer.

翻译：在表格数据的机器学习中，表格Transformer（TabTransformer）是一种先进的神经网络模型，而差分隐私（DP）是确保数据隐私的关键组件。本文研究了在迁移学习场景下将这两方面结合的优势——即对TabTransformer进行差分隐私预训练与微调，并采用多种参数高效微调（PEFT）方法，包括Adapter、LoRA和Prompt Tuning。我们在ACSIncome数据集上的大量实验表明，这些PEFT方法在下游任务准确率和可训练参数数量方面均优于传统方法，从而在参数效率、隐私保护和准确率之间实现了更优的平衡。我们的代码可从github.com/IBM/DP-TabTransformer获取。