Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs. However, achieving high-quality generation in distilled models requires careful joint design of both the student architecture and the distillation process. Many prior distillation works evaluate downstream multiple-choice benchmarks by ranking candidate answers with log-likelihood rather than requiring autoregressive generation, which can obscure important differences in model quality. For example, we show that a 7B parameter distilled model that nearly matches its teacher to within 0.2\,pp under log-likelihood scoring actually falls behind by 20.8\,pp when the model must generate answers autoregressively. We propose a Hybrid Kimi Delta Attention (Hybrid-KDA) architecture paired with GenDistill, a multi-stage distillation pipeline, and use generation-based evaluation throughout to guide design decisions. Applying this approach to Qwen3-0.6B, we systematically ablate six design axes: training objective, loss masking, training duration, dataset selection, parameter freezing, and architecture choice. We find that log-likelihood-based evaluation consistently underestimates the gap between teacher and student, and can in some cases reverse the ranking of design choices, meaning that conclusions drawn from perplexity-only evaluation may be misleading. Among the factors we study, dataset selection, completion-only masking, and freezing attention layers during post-training have the largest impact on generation quality. Our best Hybrid-KDA model retains 86--90\% of teacher accuracy on knowledge benchmarks while reducing KV cache memory by up to 75\% and improving time-to-first-token by 2--4$\times$ at 128K-token contexts.

翻译：通过蒸馏将预训练Transformer转化为更高效的混合模型是降低推理成本的有效途径。然而，在蒸馏模型中实现高质量生成需要同时精心设计学生架构与蒸馏过程。许多既往蒸馏研究通过对数似然进行候选答案排序来评估下游多项选择基准，而非要求自回归生成，这容易掩盖模型质量的关键差异。例如，我们证明：一个在逐令牌对数似然评分下与教师模型几乎持平（差距0.2个百分点）的7B参数蒸馏模型，当需要自回归生成答案时实际落后20.8个百分点。为此，我们提出混合Kimi Delta注意力（Hybrid-KDA）架构，配合多阶段蒸馏流水线GenDistill，并在整个过程中采用基于生成的评估指导设计决策。将该方法应用于Qwen3-0.6B时，我们系统性地消融了六个设计维度：训练目标、损失掩码、训练时长、数据集选择、参数冻结与架构选择。研究发现基于对数似然的评估始终低估师生模型差距，甚至在某些情况下会反转设计选择的排序——这意味着仅依赖困惑度得出的结论可能具有误导性。在所研究的因素中，数据集选择、仅补全掩码以及后训练期间冻结注意力层对生成质量影响最大。我们的最佳Hybrid-KDA模型在知识基准上保留教师模型86-90%的准确率，同时将KV缓存内存减少高达75%，并在128K令牌上下文下将首令牌生成时间提升2-4倍。