We introduce CompTok, a training framework for learning visual tokenizers whose tokens are enhanced for compositionality. CompTok uses a token-conditioned diffusion decoder. By employing an InfoGAN-style objective, where we train a recognition model to predict the tokens used to condition the diffusion decoder using the decoded images, we enforce the decoder to not ignore any of the tokens. To promote compositional control, besides the original images, CompTok also trains on tokens formed by swapping token subsets between images, enabling more compositional control of the token over the decoder. As the swapped tokens between images do not have ground truth image targets, we apply a manifold constraint via an adversarial flow regularizer to keep unpaired swap generations on the natural-image distribution. The resulting tokenizer not only achieves state-of-the-art performance on image class-conditioned generation, but also demonstrates properties such as swapping tokens between images to achieve high level semantic editing of an image. Additionally, we propose two metrics that measures the landscape of the token space that can be useful to describe not only the compositionality of the tokens, but also how easy to learn the landscape is for a generator to be trained on this space. We show in experiments that CompTok can improve on both of the metrics as well as supporting state-of-the-art generators for class conditioned generation.

翻译：本文提出CompTok，一种用于学习视觉分词器的训练框架，其分词结果在组合性方面得到增强。CompTok采用基于分词的扩散解码器。通过应用InfoGAN风格的目标函数——训练识别模型根据解码图像预测用于调节扩散解码器的分词——我们强制解码器不忽略任何分词。为提升组合控制能力，除原始图像外，CompTok还在通过交换图像间分词子集构成的分词上进行训练，从而增强分词对解码器的组合控制能力。由于图像间交换的分词缺乏真实图像目标，我们通过对抗流正则化器施加流形约束，使未配对的交换生成结果保持在自然图像分布上。所得分词器不仅在图像类别条件生成任务中达到最先进性能，还展现出通过交换图像间分词实现图像高层语义编辑的特性。此外，我们提出两个度量指标，用于刻画分词空间的拓扑结构，这些指标不仅能描述分词的组合性，还能反映生成器在该空间训练的难易程度。实验表明，CompTok能同时改进这两个指标，并支持最先进的类别条件生成器。