Variational autoencoders (VAEs) are popular models for representation learning but their encoders are susceptible to overfitting (Cremer et al., 2018) because they are trained on a finite training set instead of the true (continuous) data distribution $p_{\mathrm{data}}(\mathbf{x})$. Diffusion models, on the other hand, avoid this issue by keeping the encoder fixed. This makes their representations less interpretable, but it simplifies training, enabling accurate and continuous approximations of $p_{\mathrm{data}}(\mathbf{x})$. In this paper, we show that overfitting encoders in VAEs can be effectively mitigated by training on samples from a pre-trained diffusion model. These results are somewhat unexpected as recent findings (Alemohammad et al., 2023; Shumailov et al., 2023) observe a decay in generative performance when models are trained on data generated by another generative model. We analyze generalization performance, amortization gap, and robustness of VAEs trained with our proposed method on three different data sets. We find improvements in all metrics compared to both normal training and conventional data augmentation methods, and we show that a modest amount of samples from the diffusion model suffices to obtain these gains.

翻译：变分自编码器（VAE）是表示学习的流行模型，但其编码器容易过拟合（Cremer等，2018），这是因为它们是在有限训练集而非真实（连续）数据分布$p_{\mathrm{data}}(\mathbf{x})$上训练的。相比之下，扩散模型通过保持编码器固定来避免这一问题。这使得它们的表示可解释性较弱，但简化了训练过程，能够对$p_{\mathrm{data}}(\mathbf{x})$进行准确且连续的近似。本文表明，通过在预训练扩散模型生成的样本上训练，可以有效缓解VAE中编码器的过拟合问题。这一结果有些出人意料，因为近期研究（Alemohammad等，2023；Shumailov等，2023）发现，当模型使用另一个生成模型生成的数据进行训练时，生成性能会出现下降。我们分析了使用所提方法在三个不同数据集上训练的VAE的泛化性能、摊销差距和鲁棒性。与常规训练和传统数据增强方法相比，我们发现所有指标均有改善，并且证明只需使用扩散模型生成的适量样本即可获得这些提升。