Transfer learning on tabular data is challenging due to disparate feature spaces across domains, in contrast to the homogeneous structures of image and text. Large language models (LLMs) offer a knowledge base to improve the limited effectiveness of cross-domain transfer learning for tabular data. However, LLM performance often stagnates due to subjective text prompts and the computational limitations of in-context learning. We present a novel language-to-tabular context-learning method that uses attention-specific transformer weights, enabling seamless transfer learning across disparate tabular data sets. The LLM attention transplant mechanism facilitates a domain-agnostic transfer learning, eliminating the need for shared features between tables, LLM prompt engineering, and large-scale pretrained models. Our experiments using ten pairs of disjoint source-target data sets and 12 baseline methods demonstrate the superiority of the proposed LLM-attention transplant for transfer learning (LATTLE) method over traditional ML models, state-of-the-art deep tabular architectures, and models trained on thousands to billions of tabular samples. The proposed cross-domain attention transfer demonstrates an effective solution for adapting LLMs to learning non-text tabular data in a low-resource environment. The source code of the LATTLE implementation is publicly available.

翻译：表格数据的迁移学习因不同领域特征空间存在差异而面临挑战，这与图像和文本数据的同质化结构形成鲜明对比。大型语言模型（LLM）为改善表格数据跨领域迁移学习效果有限的问题提供了知识基础。然而，LLM的性能常因主观文本提示和上下文学习的计算限制而陷入停滞。本文提出一种新颖的语言到表格上下文学习方法，该方法利用注意力特定的Transformer权重，实现跨异构表格数据集的无缝迁移学习。LLM注意力移植机制促进了领域无关的迁移学习，无需表格间共享特征、LLM提示工程或大规模预训练模型。我们在十对互斥的源-目标数据集上使用12种基线方法进行实验，结果表明所提出的LLM注意力移植迁移学习方法（LATTLE）相较于传统机器学习模型、最先进的深度表格架构以及基于数千至数十亿表格样本训练的模型具有显著优势。所提出的跨领域注意力迁移机制为LLM在低资源环境下适应非文本表格数据学习提供了有效解决方案。LATTLE实现的源代码已公开。