While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.

翻译：尽管近年来大型语言模型（LLM）的进展显著提升了合成文本数据的质量，但表格数据的合成研究相对较少。本文通过Tabby模型弥补这一差距，该模型是对标准Transformer语言模型架构的一种简单而强大的训练后改进，使其能够用于表格数据集合成。Tabby通过门控混合专家机制配合列专属参数集，实现了对列间差异的表征。实验表明，Tabby生成的数据质量接近甚至等同于真实数据。将我们提出的新型LLM表格训练技术Plain与Tabby结合后，数据质量较现有方法最高可提升44%。研究还证明Tabby可扩展至更通用的结构化数据，在嵌套JSON数据集上同样达到了与真实数据相当的性能水平。