Variational autoencoders (VAEs) are popular models for representation learning but their encoders are susceptible to overfitting (Cremer et al., 2018) because they are trained on a finite training set instead of the true (continuous) data distribution $p_{\mathrm{data}}(\mathbf{x})$. Diffusion models, on the other hand, avoid this issue by keeping the encoder fixed. This makes their representations less interpretable, but it simplifies training, enabling accurate and continuous approximations of $p_{\mathrm{data}}(\mathbf{x})$. In this paper, we show that overfitting encoders in VAEs can be effectively mitigated by training on samples from a pre-trained diffusion model. These results are somewhat unexpected as recent findings (Alemohammad et al., 2023; Shumailov et al., 2023) observe a decay in generative performance when models are trained on data generated by another generative model. We analyze generalization performance, amortization gap, and robustness of VAEs trained with our proposed method on three different data sets. We find improvements in all metrics compared to both normal training and conventional data augmentation methods, and we show that a modest amount of samples from the diffusion model suffices to obtain these gains.

翻译：变分自编码器（VAEs）是表征学习中常用的模型，但其编码器容易过拟合（Cremer et al., 2018），因为它们在有限训练集而非真实（连续）数据分布 $p_{\mathrm{data}}(\mathbf{x})$ 上训练。相比之下，扩散模型通过保持编码器固定来避免这一问题。这使得其表征可解释性较差，但简化了训练，能够准确且连续地近似 $p_{\mathrm{data}}(\mathbf{x})$。本文表明，在预训练的扩散模型生成的样本上训练，可以有效缓解VAE中编码器的过拟合问题。这一结果有些出人意料，因为近期研究（Alemohammad et al., 2023；Shumailov et al., 2023）观察到，当模型在另一个生成模型生成的数据上训练时，生成性能会下降。我们分析了使用所提方法在三个不同数据集上训练的VAE的泛化性能、摊销间隙和鲁棒性。与常规训练和传统数据增强方法相比，我们发现在所有指标上均有改进，并且扩散模型生成的少量样本足以获得这些增益。