Learning image classification and image generation using the same set of network parameters is a challenging problem. Recent advanced approaches perform well in one task often exhibit poor performance in the other. This work introduces an energy-based classifier and generator, namely EGC, which can achieve superior performance in both tasks using a single neural network. Unlike a conventional classifier that outputs a label given an image (i.e., a conditional distribution $p(y|\mathbf{x})$), the forward pass in EGC is a classifier that outputs a joint distribution $p(\mathbf{x},y)$, enabling an image generator in its backward pass by marginalizing out the label $y$. This is done by estimating the energy and classification probability given a noisy image in the forward pass, while denoising it using the score function estimated in the backward pass. EGC achieves competitive generation results compared with state-of-the-art approaches on ImageNet-1k, CelebA-HQ and LSUN Church, while achieving superior classification accuracy and robustness against adversarial attacks on CIFAR-10. This work represents the first successful attempt to simultaneously excel in both tasks using a single set of network parameters. We believe that EGC bridges the gap between discriminative and generative learning.

翻译：使用同一组网络参数同时学习图像分类与图像生成是一项具有挑战性的问题。近年来取得优异性能的先进方法往往在某一任务上表现突出，而在另一任务上性能不佳。本文提出了一种基于能量的分类器与生成器，即EGC，该模型能够通过单一神经网络在这两项任务上均实现卓越性能。与传统分类器根据输入图像输出标签（即条件分布$p(y|\mathbf{x})$）不同，EGC的前向传播输出的是联合分布$p(\mathbf{x},y)$的分类器，从而通过在后向传播中边缘化标签$y$实现图像生成功能。具体而言，前向过程通过估计带噪图像的能量与分类概率，而后向过程则利用估计得到的得分函数进行去噪。在ImageNet-1k、CelebA-HQ和LSUN Church数据集上，EGC实现了与现有最优方法相媲美的生成结果，同时在CIFAR-10数据集上展现出卓越的分类准确率与对抗攻击鲁棒性。本文首次成功实现了使用单一网络参数集合同时在这两项任务上达到顶尖性能。我们相信EGC弥合了判别式学习与生成式学习之间的鸿沟。