Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical RePresentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5% conversion improvement in the first half after its initial launch, and +1.03% and +1.22% conversion improvement from two individual launches in the subsequent half.

翻译：知识蒸馏（KD）将大型基础模型（FM）的单一标量预测迁移至紧凑型垂直模型（VM），但受限于迁移比率递减——即VM捕获的FM性能提升比例——因为单一标量无法传递大型FM所学习的丰富中间知识。为解决这一瓶颈，我们提出LoopFM（学习FM的历史表示），一种通过将FM中间嵌入作为结构化输入特征（如用户历史序列）提供给下游VM的高带宽迁移通道框架，且无需在服务阶段进行实时FM推理，亦无需FM与VM之间的架构耦合。我们为LoopFM提供了包含增益分解和迁移比率分析的理论框架。在三个公开基准测试中，LoopFM展示了显著的AUC提升（例如在淘宝广告数据集上提升6%以上）以及与KD互补的知识迁移能力。在工业级系统（数十亿样本、万亿参数FM）中，LoopFM在KD基础上将知识迁移比率近乎翻倍，首次上线后上半年转化率提升0.5%，下半年两次独立上线分别带来1.03%和1.22%的转化率提升。