Multimodal LLMs use dedicated encoders to bridge non-language modalities (vision encoders for images, depth models for audio codec tokens) because raw token embeddings alone cannot capture modality-specific structure. We argue that Semantic IDs (SIDs), the hierarchical codes used in generative recommendation, constitute another such modality: a SID level token's meaning depends on its prefix context, yet current systems simply add SID tokens to the vocabulary and rely on training to learn these context-dependent meanings from scratch. We propose PrefixMem, a lightweight SID encoder based on prefix n-gram memory tables that provides the LLM with structured, prefix-conditioned representations at SID token positions. Like vision encoders in multimodal LLMs, PrefixMem can be pre-trained independently and then attached to any LLM for joint training. We evaluate on large-scale data from Pinterest across multiple LLM families and show that PrefixMem improves deepest-level SID accuracy by up to 46% relative and full-SID retrieval recall by up to 22% relative at matched training compute. The encoder's benefit concentrates on hard examples where greedy decoding fails, with up to 77% relative accuracy gains, confirming that SID tokens benefit from a dedicated encoder just as other non-language modalities do.

翻译：多模态大语言模型（Multimodal LLMs）通过专用编码器桥接非语言模态（如图像编码器、音频编解码器令牌的深度模型），因为原始令牌嵌入无法单独捕捉模态特定结构。我们认为语义ID（Semantic IDs, SIDs）——生成式推荐中使用的层次化编码——构成了另一种此类模态：SID层级令牌的含义取决于其前缀上下文，而当前系统仅将SID令牌加入词汇表，并依赖训练从头学习这些上下文相关含义。我们提出PrefixMem，一种基于前缀n-gram记忆表的轻量级SID编码器，该编码器在LLM的SID令牌位置提供结构化的、受前缀条件约束的表示。与多模态LLM中的图像编码器类似，PrefixMem可独立预训练，随后接入任意LLM进行联合训练。我们在Pinterest的大规模数据上，跨多个LLM家族评估表明：在相同训练算力下，PrefixMem将最深层级SID准确率相对提升高达46%，完整SID检索召回率相对提升高达22%。该编码器的优势集中于贪婪解码失败的困难样本，准确率相对增益高达77%，这验证了SID令牌与其他非语言模态一样，需要专用编码器的支持。