Image recognition and generation have long been developed independently of each other. With the recent trend towards general-purpose representation learning, the development of general representations for both recognition and generation tasks is also promoted. However, preliminary attempts mainly focus on generation performance, but are still inferior on recognition tasks. These methods are modeled in the vector-quantized (VQ) space, whereas leading recognition methods use pixels as inputs. Our key insights are twofold: (1) pixels as inputs are crucial for recognition tasks; (2) VQ tokens as reconstruction targets are beneficial for generation tasks. These observations motivate us to propose an Alternating Denoising Diffusion Process (ADDP) that integrates these two spaces within a single representation learning framework. In each denoising step, our method first decodes pixels from previous VQ tokens, then generates new VQ tokens from the decoded pixels. The diffusion process gradually masks out a portion of VQ tokens to construct the training samples. The learned representations can be used to generate diverse high-fidelity images and also demonstrate excellent transfer performance on recognition tasks. Extensive experiments show that our method achieves competitive performance on unconditional generation, ImageNet classification, COCO detection, and ADE20k segmentation. Importantly, our method represents the first successful development of general representations applicable to both generation and dense recognition tasks. Code shall be released.

翻译：图像识别与生成长期独立发展。随着近期通用表征学习趋势的兴起，推动面向识别与生成任务的通用表征发展也成为关注焦点。然而，初步尝试主要侧重生成性能，在识别任务上仍存在不足。这些方法基于向量量化空间建模，而主流识别方法以像素为输入。我们的关键发现有二：（1）像素作为输入对识别任务至关重要；（2）VQ令牌作为重建目标对生成任务大有裨益。基于此，我们提出交替去噪扩散过程，在单一表征学习框架内融合这两个空间。在每个去噪步骤中，本方法首先从先前VQ令牌解码像素，再从解码像素生成新VQ令牌。扩散过程逐步掩蔽部分VQ令牌以构建训练样本。学习到的表征既能生成多样高保真图像，也在识别任务中展现出优异的迁移性能。大量实验表明，本方法在无条件生成、ImageNet分类、COCO检测和ADE20k分割任务上均取得竞争性表现。值得注意的是，本方法首次成功开发出可同时适用于生成与密集识别任务的通用表征。代码将公开发布。