We introduce generative infinite-vocabulary transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a VAE. When applying GIVT to class-conditional image generation with iterative masked modeling, we show competitive results with MaskGIT, while our approach outperforms both VQ-GAN and MaskGIT when using it for causal modeling. Finally, we obtain competitive results outside of image generation when applying our approach to panoptic segmentation and depth estimation with a VAE-based variant of the UViM framework.

翻译：我们提出生成式无限词汇Transformer（GIVT），该模型可生成实数值向量序列，而非来自有限词汇表的离散词元。为此，我们引入两项针对仅有解码器Transformer的简洁改进：1）在输入层，用输入向量的线性投影替代有限词汇表查询表；2）在输出层，将通常映射为分类分布的logits预测替换为多元高斯混合模型的参数。受VQ-GAN与MaskGIT图像生成范式的启发（该范式使用Transformer建模VQ-VAE的离散潜在序列），我们采用GIVT对VAE的未量化实值潜在序列进行建模。当将GIVT应用于基于迭代掩码建模的类别条件图像生成时，其性能与MaskGIT相当；而在因果建模场景下，本方法全面超越VQ-GAN与MaskGIT。最后，我们将该方法应用于基于VAE变体UViM框架的全景分割与深度估计任务，在图像生成领域之外亦取得具有竞争力的结果。