Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and commit to open research by first releasing inference code and model weights at https://github.com/soda-inria/tabicl, with synthetic data engine and pretraining code to follow.

翻译：表格基础模型，如TabPFNv2和TabICL，近期已在预测基准测试中超越梯度提升树，展示了上下文学习在表格数据上的价值。我们推出TabICLv2，这是一个基于三大支柱构建的、用于回归和分类任务的新一代最先进基础模型：（1）一种专为高预训练多样性设计的新型合成数据生成引擎；（2）多项架构创新，包括一种新的可扩展注意力softmax，可在无需进行代价高昂的长序列预训练的情况下，提升模型对更大数据集的泛化能力；（3）优化的预训练协议，特别是用Muon优化器替代AdamW。在TabArena和TALENT基准测试中，未经任何调优的TabICLv2超越了当前最先进模型RealTabPFN-2.5（在真实数据上进行了超参数调优、集成和微调）的性能。仅使用中等规模的预训练计算资源，TabICLv2就能在50GB GPU内存限制下有效泛化至百万级规模的数据集，同时速度显著快于RealTabPFN-2.5。我们提供了详尽的消融研究以量化这些贡献，并承诺开放研究：首先在https://github.com/soda-inria/tabicl 发布推理代码和模型权重，合成数据引擎和预训练代码将随后发布。