In this paper, we propose a novel generative model that utilizes a conditional Energy-Based Model (EBM) for enhancing Variational Autoencoder (VAE), termed Energy-Calibrated VAE (EC-VAE). Specifically, VAEs often suffer from blurry generated samples due to the lack of a tailored training on the samples generated in the generative direction. On the other hand, EBMs can generate high-quality samples but require expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we introduce a conditional EBM for calibrating the generative direction of VAE during training, without requiring it for the generation at test time. In particular, we train EC-VAE upon both the input data and the calibrated samples with adaptive weight to enhance efficacy while avoiding MCMC sampling at test time. Furthermore, we extend the calibration idea of EC-VAE to variational learning and normalizing flows, and apply EC-VAE to an additional application of zero-shot image restoration via neural transport prior and range-null theory. We evaluate the proposed method with two applications, including image generation and zero-shot image restoration, and the experimental results show that our method achieves the state-of-the-art performance over single-step non-adversarial generation. Our code is available at https://github.com/DJ-LYH/EC-VAE.

翻译：本文提出一种新型生成模型，利用条件能量基模型（EBM）增强变分自编码器（VAE），称为能量校准变分自编码器（EC-VAE）。具体而言，VAE因缺乏对生成方向样本的定制化训练，常导致生成样本模糊；而EBM虽能生成高质量样本，但需要昂贵的马尔可夫链蒙特卡洛（MCMC）采样。为解决这些问题，我们在训练阶段引入条件EBM校准VAE的生成方向，且无需在测试阶段使用该EBM进行生成。特别地，我们基于输入数据和经自适应权重校准的样本共同训练EC-VAE，在提升效能的同时避免测试时的MCMC采样。进一步，我们将EC-VAE的校准思想扩展到变分学习与归一化流，并基于神经传输先验与零空间理论，将EC-VAE应用于零样本图像复原任务。通过图像生成与零样本图像复原两项应用验证，实验结果表明该方法在单步非对抗生成中达到了最优性能。相关代码已在https://github.com/DJ-LYH/EC-VAE开源。