We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods. TabPFN is fully entailed in the weights of our network, which accepts training and test samples as a set-valued input and yields predictions for the entire test set in a single forward pass. TabPFN is a Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior. This prior incorporates ideas from causal reasoning: It entails a large space of structural causal models with a preference for simple structures. On the 18 datasets in the OpenML-CC18 suite that contain up to 1 000 training data points, up to 100 purely numerical features without missing values, and up to 10 classes, we show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 70$\times$ speedup. This increases to a 3200$\times$ speedup when a GPU is available. We also validate these results on an additional 67 small numerical datasets from OpenML. We provide all our code, the trained TabPFN, an interactive browser demo and a Colab notebook at https://github.com/automl/TabPFN.

翻译：我们提出TabPFN——一个经过训练的Transformer模型，能够在不到一秒钟内完成小型表格数据集的监督分类任务，无需超参数调优，且性能可与最先进的分类方法相媲美。TabPFN完全内嵌于网络权重中，该网络将训练样本和测试样本作为集合值输入，通过单次前向传播即可对整个测试集进行预测。TabPFN是一种先验数据拟合网络（PFN），仅需离线训练一次，用于近似基于从我们设计的先验分布中生成的合成数据集的贝叶斯推断。该先验借鉴了因果推理思想：它包含一个庞大的结构因果模型空间，并倾向于简单结构。在OpenML-CC18套件中18个包含最多1000个训练数据点、最多100个纯数值特征（无缺失值）以及最多10个类别的数据集上，我们证明该方法明显优于提升树模型，且性能与复杂的先进AutoML系统相当，同时实现高达70倍的加速。当使用GPU时，加速比可提升至3200倍。我们还在OpenML的另外67个小型数值数据集上验证了这些结果。所有代码、训练好的TabPFN、交互式浏览器演示及Colab笔记本均已开源：https://github.com/automl/TabPFN