Recent advancements in NLP have witnessed the groundbreaking impact of pretrained models, yielding impressive outcomes across various tasks. This study seeks to extend the power of pretraining methodologies to facilitating the prediction over tables in data science, a domain traditionally overlooked, yet inherently challenging due to the plethora of table schemas intrinsic to different tasks. The primary research questions underpinning this work revolve around the establishment of a universal pretraining protocol for tables with varied structures, the generalizability and transferability of learned knowledge across tasks, the adaptation to diverse downstream applications, and the incorporation of incremental columns over time. In response to these challenges, we introduce UniTabE, a straightforward yet effective method designed to process tables in a uniform manner, devoid of constraints imposed by specific table structures. UniTabE's core concept relies on representing each basic table element with a module, termed TabUnit. This is subsequently followed by a Transformer encoder to refine the representation. Moreover, our model is designed to facilitate pretraining and finetuning through the utilization of free-form prompts. In order to implement the pretraining phase, we curated an expansive tabular dataset comprising approximately 13B samples, meticulously gathered from the Kaggle platform. This research primarily centers on classification and regression tasks involving tabular data, and conducts rigorous experimental testing and analyses to validate the effectiveness of our methodology. The experimental results demonstrate UniTabE's superior performance against several baselines across massive benchmarks. This, therefore, underscores UniTabE's potential to significantly enhance the semantic representation of tabular data, thereby marking a significant stride for tabular data analysis.

翻译：自然语言处理领域的最新进展见证了预训练模型的突破性影响，其在各类任务中取得了令人瞩目的成果。本研究旨在将预训练方法的能力拓展至数据科学中的表格预测，这一领域虽长期被忽视，却因不同任务所固有的表格模式多样性而充满挑战。本工作的核心研究问题围绕构建适用于不同结构表格的通用预训练协议、所学知识的跨任务泛化性与可迁移性、对多样化下游应用的适应能力，以及随时间推移的增量列整合展开。为应对这些挑战，我们提出UniTabE——一种简洁而有效的方法，能够以统一方式处理表格，不受特定表格结构的约束。UniTabE的核心思想在于利用名为TabUnit的模块表示每个基本表格元素，随后通过Transformer编码器优化其表示。此外，我们的模型设计支持基于自由形式提示的预训练与微调。为实施预训练阶段，我们构建了包含约130亿样本的大规模表格数据集，这些数据精心收集自Kaggle平台。本研究主要聚焦于表格数据的分类与回归任务，并进行了严格的实验测试与分析以验证方法的有效性。实验结果表明，UniTabE在多个基准测试中相较于多种基线方法表现出优越性能。因此，这凸显了UniTabE在显著增强表格数据语义表示方面的潜力，标志着表格数据分析迈出了重要一步。