Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships between features, especially when handling mixed data types. In contrast, Transformers, through their attention mechanism, are better suited for capturing complex feature interactions. In this paper, we empirically investigate the impact of integrating Transformers into different components of a VAE. We conduct experiments on 57 datasets from the OpenML CC18 suite and draw two main conclusions. First, results indicate that positioning Transformers to leverage latent and decoder representations leads to a trade-off between fidelity and diversity. Second, we observe a high similarity between consecutive blocks of a Transformer in all components. In particular, in the decoder, the relationship between the input and output of a Transformer is approximately linear.

翻译：表格数据对于生成模型而言仍是一个具有挑战性的领域。具体来说，标准的变分自编码器（VAE）架构通常由多层感知机构成，难以对特征间的关系进行建模，尤其是在处理混合数据类型时。相比之下，Transformer凭借其注意力机制，更擅长捕捉复杂的特征交互。本文通过实验研究了将Transformer集成到VAE不同组件中的影响。我们在OpenML CC18套件的57个数据集上进行了实验，并得出两个主要结论。首先，结果表明，将Transformer置于利用潜在表示和解码器表示的位置会导致保真度与多样性之间的权衡。其次，我们观察到Transformer在所有组件中连续块之间具有高度相似性。特别是在解码器中，Transformer的输入与输出之间的关系近似为线性。