In this paper, we propose a novel generative model that utilizes a conditional Energy-Based Model (EBM) for enhancing Variational Autoencoder (VAE), termed Energy-Calibrated VAE (EC-VAE). Specifically, VAEs often suffer from blurry generated samples due to the lack of a tailored training on the samples generated in the generative direction. On the other hand, EBMs can generate high-quality samples but require expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we introduce a conditional EBM for calibrating the generative direction of VAE during training, without requiring it for the generation at test time. In particular, we train EC-VAE upon both the input data and the calibrated samples with adaptive weight to enhance efficacy while avoiding MCMC sampling at test time. Furthermore, we extend the calibration idea of EC-VAE to variational learning and normalizing flows, and apply EC-VAE to an additional application of zero-shot image restoration via neural transport prior and range-null theory. We evaluate the proposed method with two applications, including image generation and zero-shot image restoration, and the experimental results show that our method achieves the state-of-the-art performance over single-step non-adversarial generation.

翻译：本文提出一种利用条件能量基模型增强变分自编码器的新型生成模型，称为能量校准变分自编码器。具体而言，变分自编码器常因生成方向缺乏针对性训练而导致样本模糊；而能量基模型虽能生成高质量样本，但需耗费高昂的马尔可夫链蒙特卡洛采样。为解决上述问题，我们在训练阶段引入条件能量基模型校准变分自编码器的生成方向，且测试时无需该模型参与生成。具体地，我们通过自适应权重对输入数据与校准样本联合训练能量校准变分自编码器，在避免测试时马尔可夫链蒙特卡洛采样的同时提升效能。此外，我们将能量校准变分自编码器的校准思想拓展至变分学习与归一化流领域，并基于神经传输先验与零空间理论将其应用于零样本图像修复任务。通过图像生成与零样本图像修复两项应用评估，实验结果表明该方法在单步非对抗生成任务中达到当前最优性能。