Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers (IDs). A fundamental yet unexplored design choice is that existing methods employ fixed-length tokenization for all items, implicitly assuming uniform encoding capacity regardless of item characteristics. Through systematic experiments across four datasets, we discover the Popularity-Length Paradox: popular items achieve optimal performance with short IDs, while tail items require substantially longer codes to capture discriminative semantics. This reveals a critical mismatch where popular items benefit from abundant collaborative signals and require minimal semantic detail, whereas tail items must rely on fine-grained content features due to sparse interaction data. To address this, we propose VarLenRec, a framework for learning variable-length tokenization. We develop Popularity-Weighted Information Budget Allocation (PIBA), an information-theoretic framework proving that optimal ID length should scale as a negative power of popularity. Directly implementing variable-length allocation faces two technical challenges: standard Euclidean residual quantization lacks geometric capacity to support diverse code lengths without distortion, and discrete length decisions are non-differentiable. We address these through Hyperbolic Residual Quantization, which leverages the exponential volume growth of the Poincaré ball to naturally stratify encoding capacity, and a Soft Length Controller, which enables differentiable length prediction via continuous layer retention probabilities regularized by PIBA-derived priors. Extensive experiments demonstrate that VarLenRec achieves significant improvements over state-of-the-art methods in recommendation accuracy and training/inference efficiency, revealing the importance of adaptive encoding capacity in generative recommendation.

翻译：生成式推荐将推荐问题重新构想为对离散语义标识符的下一词元预测。现有方法中一个根本但未被探索的设计选择是：所有物品均采用固定长度分词，这隐含地假设了无论物品特性如何，编码容量均保持一致。通过在四个数据集上进行系统性实验，我们发现“流行度-长度悖论”：热门物品使用短标识符即可达到最优性能，而长尾物品则需要显著更长的编码来捕获区分性语义。这揭示了一个关键不匹配：热门物品受益于丰富的协同信号且仅需极简语义细节，而长尾物品由于交互数据稀疏必须依赖细粒度内容特征。为解决这一问题，我们提出VarLenRec——一个学习可变长度分词的框架。我们提出流行度加权信息预算分配（PIBA），这是一个信息论框架，证明最优标识符长度应与流行度的负幂次方成比例。直接实现可变长度分配面临两大技术挑战：标准欧几里得残差量化缺乏几何容量支持不同编码长度而不失真，且离散长度决策不可微。我们通过双曲残差量化（利用庞加莱球的指数级体积增长自然分层编码容量）和软长度控制器（通过连续层保留概率实现可微长度预测，并由PIBA先验正则化）解决这些问题。大量实验表明，VarLenRec在推荐准确性和训练/推理效率上均显著优于现有最先进方法，揭示了生成式推荐中自适应编码容量的重要性。