Learning image classification and image generation using the same set of network parameters is a challenging problem. Recent advanced approaches perform well in one task often exhibit poor performance in the other. This work introduces an energy-based classifier and generator, namely EGC, which can achieve superior performance in both tasks using a single neural network. Unlike a conventional classifier that outputs a label given an image (i.e., a conditional distribution $p(y|\mathbf{x})$), the forward pass in EGC is a classifier that outputs a joint distribution $p(\mathbf{x},y)$, enabling an image generator in its backward pass by marginalizing out the label $y$. This is done by estimating the energy and classification probability given a noisy image in the forward pass, while denoising it using the score function estimated in the backward pass. EGC achieves competitive generation results compared with state-of-the-art approaches on ImageNet-1k, CelebA-HQ and LSUN Church, while achieving superior classification accuracy and robustness against adversarial attacks on CIFAR-10. This work represents the first successful attempt to simultaneously excel in both tasks using a single set of network parameters. We believe that EGC bridges the gap between discriminative and generative learning.

翻译：使用同一网络参数同时学习图像分类与图像生成是一项具有挑战性的课题。近期先进方法虽能在单一任务中表现优异，但往往在另一任务中性能欠佳。本文提出一种基于能量的分类生成器EGC，其通过单一神经网络即可在两个任务中实现卓越性能。不同于传统分类器根据图像输出标签（即条件分布$p(y|\mathbf{x})$），EGC的前向传播作为分类器输出联合分布$p(\mathbf{x},y)$，并通过反向传播中对标签$y$进行边缘化处理实现图像生成。具体而言，前向传播通过估计带噪图像的能量与分类概率，反向传播则利用所估计的分数函数对图像进行去噪。在ImageNet-1k、CelebA-HQ和LSUN Church数据集上，EGC获得了与当前最优方法相当的生成结果；在CIFAR-10数据集上，其分类精度与对抗攻击鲁棒性均表现优越。本研究首次成功实现了使用单一网络参数集同时精通两项任务，我们相信EGC弥合了判别式学习与生成式学习之间的鸿沟。