Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is still suboptimal as they sample multiple tokens simultaneously without considering the dependence among them. We empirically investigate this problem and propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. To further improve the image quality, we introduce a cohesive sampling strategy, Frequency Adaptive Sampling (FAS), to each group of tokens divided according to the self-attention maps. We validate the efficacy of TCTS combined with FAS with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality. Our text-conditioned sampling framework further reduces the original inference time by more than 50% without modifying the original generative model.

翻译：基于词元的掩码生成模型因其并行解码带来的快速推理速度而日益流行。尽管近期基于词元的方法在性能上已达到与扩散模型相媲美的水平，但其生成性能仍不理想，原因在于同时采样多个词元时未能考虑它们之间的依赖关系。我们通过实验研究这一问题，并提出一种可学习的采样模型——文本条件化词元选择（TCTS），利用文本信息通过局部监督来选取最优词元。TCTS不仅提升了生成图像的视觉质量，还增强了其与给定文本的语义对齐。为了进一步改善图像质量，我们引入了一种协同采样策略——频率自适应采样（FAS），将其应用于根据自注意力图划分的每组词元。我们在多种生成任务中验证了TCTS与FAS组合的有效性，结果表明，在图像-文本对齐和图像质量方面，该方法显著优于基线模型。我们的文本条件化采样框架在不修改原始生成模型的情况下，将原始推理时间进一步缩短50%以上。