We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a $\beta$-VAE. In class-conditional image generation GIVT outperforms VQ-GAN (and improved variants thereof) as well as MaskGIT, and achieves performance competitive with recent latent diffusion models. Finally, we obtain strong results outside of image generation when applying GIVT to panoptic segmentation and depth estimation with a VAE variant of the UViM framework.

翻译：我们提出了生成式无限词汇Transformer（GIVT），该模型生成具有实值分量的向量序列，而非来自有限词汇表的离散标记。为此，我们对仅含解码器的Transformer提出了两个出奇简单的修改：1）在输入层，我们将有限词汇查找表替换为输入向量的线性投影；2）在输出层，我们将通常映射为分类分布的对数几率预测替换为多元高斯混合模型的参数。受VQ-GAN和MaskGIT的图像生成范式启发（其中Transformer用于建模VQ-VAE的离散潜在序列），我们使用GIVT来建模$\beta$-VAE的未量化实值潜在序列。在类别条件图像生成任务中，GIVT的表现优于VQ-GAN（及其改进变体）和MaskGIT，并实现了与近期潜在扩散模型相竞争的性能。最后，当将GIVT应用于UViM框架的VAE变体进行全景分割和深度估计时，我们在图像生成之外的领域也取得了显著成果。