Flexible image tokenizers aim to represent an image using an ordered 1D variable-length token sequence. This flexible tokenization is typically achieved through nested dropout, where a portion of trailing tokens is randomly truncated during training, and the image is reconstructed using the remaining preceding sequence. However, this tail-truncation strategy inherently concentrates the image information in the early tokens, limiting the effectiveness of downstream AutoRegressive (AR) image generation as the token length increases. To overcome these limitations, we propose \textbf{ReToK}, a flexible tokenizer with \underline{Re}dundant \underline{Tok}en Padding and Hierarchical Semantic Regularization, designed to fully exploit all tokens for enhanced latent modeling. Specifically, we introduce \textbf{Redundant Token Padding} to activate tail tokens more frequently, thereby alleviating information over-concentration in the early tokens. In addition, we apply \textbf{Hierarchical Semantic Regularization} to align the decoding features of earlier tokens with those from a pre-trained vision foundation model, while progressively reducing the regularization strength toward the tail to allow finer low-level detail reconstruction. Extensive experiments demonstrate the effectiveness of ReTok: on ImageNet 256$\times$256, our method achieves superior generation performance compared with both flexible and fixed-length tokenizers. Code will be available at: \href{https://github.com/zfu006/ReTok}{https://github.com/zfu006/ReTok}

翻译：灵活图像分词器旨在使用有序的一维变长令牌序列来表示图像。这种灵活的分词通常通过嵌套丢弃实现，即在训练过程中随机截断部分尾部令牌，并使用剩余的前序序列重建图像。然而，这种尾部截断策略本质上将图像信息集中在早期令牌中，随着令牌长度增加，限制了下游自回归图像生成的有效性。为克服这些限制，我们提出了 **ReToK**，一种具有**冗**余**令牌**填充和分层语义正则化的灵活分词器，旨在充分利用所有令牌以增强潜在建模。具体而言，我们引入**冗余令牌填充**以更频繁地激活尾部令牌，从而缓解信息在早期令牌中的过度集中。此外，我们应用**分层语义正则化**，将早期令牌的解码特征与预训练视觉基础模型的特征对齐，同时逐步降低向尾部施加的正则化强度，以允许更精细的低层细节重建。大量实验证明了ReTok的有效性：在ImageNet 256$\times$256数据集上，与灵活和固定长度的分词器相比，我们的方法实现了更优的生成性能。代码将发布于：\href{https://github.com/zfu006/ReTok}{https://github.com/zfu006/ReTok}