In this paper, we propose a novel Energy-Calibrated Generative Model that utilizes a Conditional EBM for enhancing Variational Autoencoders (VAEs). VAEs are sampling efficient but often suffer from blurry generation results due to the lack of training in the generative direction. On the other hand, Energy-Based Models (EBMs) can generate high-quality samples but require expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we introduce a Conditional EBM for calibrating the generative direction during training, without requiring it for test time sampling. Our approach enables the generative model to be trained upon data and calibrated samples with adaptive weight, thereby enhancing efficiency and effectiveness without necessitating MCMC sampling in the inference phase. We also show that the proposed approach can be extended to calibrate normalizing flows and variational posterior. Moreover, we propose to apply the proposed method to zero-shot image restoration via neural transport prior and range-null theory. We demonstrate the effectiveness of the proposed method through extensive experiments in various applications, including image generation and zero-shot image restoration. Our method shows state-of-the-art performance over single-step non-adversarial generation.

翻译：本文提出了一种新颖的能量校准生成模型，该模型利用条件能量模型（Conditional EBM）来增强变分自编码器（VAE）。VAE采样效率高，但由于生成方向缺乏训练常导致模糊的生成结果。而能量模型（EBM）能生成高质量样本，但需要昂贵的马尔可夫链蒙特卡洛（MCMC）采样。为解决这些问题，我们引入条件能量模型在训练过程中校准生成方向，而测试时采样无需使用该模型。本方法使得生成模型能够基于数据和具有自适应权重的校准样本进行训练，从而在推理阶段无需MCMC采样的前提下提升效率与效果。我们同时证明该方法可扩展用于校准归一化流和变分后验。此外，我们提出通过神经传输先验和零空间理论将所提方法应用于零样本图像恢复。通过在图像生成和零样本图像恢复等多种应用中的大量实验，我们验证了该方法的有效性。本方法在单步非对抗生成任务中达到了当前最优性能。