Hybrid sequence models that combine efficient Transformer components with linear sequence modeling blocks are a promising alternative to pure Transformers, but most are still pretrained from scratch and therefore fail to reuse existing Transformer checkpoints. We study upcycling as a practical path to convert pretrained Transformer LLMs into hybrid architectures while preserving short-context quality and improving long-context capability. We call our solution \emph{HyLo} (HYbrid LOng-context): a long-context upcycling recipe that combines architectural adaptation with efficient Transformer blocks, Multi-Head Latent Attention (MLA), and linear blocks (Mamba2 or Gated DeltaNet), together with staged long-context training and teacher-guided distillation for stable optimization. HyLo extends usable context length by up to $32\times$ through efficient post-training and reduces KV-cache memory by more than $90\%$, enabling up to 2M-token prefill and decoding in our \texttt{vLLM} inference stack, while comparable Llama baselines run out of memory beyond 64K context. Across 1B- and 3B-scale settings (Llama- and Qwen-based variants), HyLo delivers consistently strong short- and long-context performance and significantly outperforms state-of-the-art upcycled hybrid baselines on long-context evaluations such as RULER. Notably, at similar scale, HyLo-Qwen-1.7B trained on only 10B tokens significantly outperforms JetNemotron (trained on 400B tokens) on GSM8K, Lm-Harness common sense reasoning and RULER-64K.

翻译：混合序列模型将高效Transformer组件与线性序列建模模块相结合，是纯Transformer的有前景替代方案，但大多数仍需从头预训练，因此无法复用现有Transformer检查点。我们研究将升级作为将预训练Transformer大语言模型转化为混合架构的实用路径，同时保持短上下文质量并提升长上下文能力。我们将解决方案命名为HyLo（混合长上下文）：一种结合架构适配（含高效Transformer模块、多头潜在注意力（MLA）和线性模块（Mamba2或门控DeltaNet））的长上下文升级方案，并采用分阶段长上下文训练与教师引导蒸馏以实现稳定优化。HyLo通过高效后训练将可用上下文长度提升达32倍，减少KV缓存内存超过90%，使我们的vLLM推理堆栈支持高达200万令牌的预填充和解码，而同等Llama基线在超过64K上下文时即耗尽内存。在1B和3B规模设置（基于Llama和Qwen的变体）下，HyLo在短上下文和长上下文任务中均表现持续强劲，并在RULER等长上下文评估中显著优于最先进的升级混合基线。值得注意的是，在类似规模下，仅用10B令牌训练的HyLo-Qwen-1.7B在GSM8K、Lm-Harness常识推理和RULER-64K任务上显著优于用400B令牌训练的JetNemotron。