In this paper, we propose a novel generative model that utilizes a conditional Energy-Based Model (EBM) for enhancing Variational Autoencoder (VAE), termed Energy-Calibrated VAE (EC-VAE). Specifically, VAEs often suffer from blurry generated samples due to the lack of a tailored training on the samples generated in the generative direction. On the other hand, EBMs can generate high-quality samples but require expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we introduce a conditional EBM for calibrating the generative direction of VAE during training, without requiring it for the generation at test time. In particular, we train EC-VAE upon both the input data and the calibrated samples with adaptive weight to enhance efficacy while avoiding MCMC sampling at test time. Furthermore, we extend the calibration idea of EC-VAE to variational learning and normalizing flows, and apply EC-VAE to an additional application of zero-shot image restoration via neural transport prior and range-null theory. We evaluate the proposed method with two applications, including image generation and zero-shot image restoration, and the experimental results show that our method achieves competitive performance over single-step non-adversarial generation. Our code is available at https://github.com/DJ-LYH/EC-VAE.

翻译：本文提出一种新型生成模型，通过引入条件能量基模型（EBM）增强变分自编码器（VAE），命名为能量校准VAE（EC-VAE）。具体而言，VAE常因生成方向缺乏对样本的针对性训练而导致生成样本模糊；而EBM可生成高质量样本，但需要昂贵的马尔可夫链蒙特卡洛（MCMC）采样。为解决这些问题，我们引入条件EBM在训练阶段校准VAE的生成方向，而无需在测试时使用该EBM进行生成。特别地，我们基于输入数据和自适应加权的校准样本联合训练EC-VAE，从而在避免测试时MCMC采样的同时提升生成效果。此外，我们将EC-VAE的校准思想扩展到变分学习和归一化流，并通过神经传输先验和零空间理论将其应用于零样本图像恢复任务。我们通过图像生成和零样本图像恢复两项应用评估该方法，实验结果表明，我们的方法在单步非对抗生成任务中取得了具有竞争力的性能。代码已开源：https://github.com/DJ-LYH/EC-VAE。