Large Language Models (LLMs) encode vast amounts of knowledge in their massive parameters, which is accessible to locate, trace, and analyze. Despite advances in neural interpretability, it is still not clear how to transfer knowledge in a fine-grained manner, namely parametric knowledge transfer (PKT). A key problem is enabling effective and efficient knowledge transfer across LLMs of different scales, which is essential for achieving greater flexibility and broader applicability in transferring knowledge between LLMs. Due to neural incompatibility, referring to the architectural and parametric differences between LLMs of varying scales, existing methods that directly reuse layer parameters are severely limited. In this paper, we identify the semantic alignment in latent space as the fundamental prerequisite for LLM cross-scale knowledge transfer. Instead of directly using the layer parameters, our approach takes activations as the medium of layer-wise knowledge transfer. Leveraging the semantics in latent space, our approach is simple and outperforms prior work, better aligning model behaviors across varying scales. Evaluations on four benchmarks demonstrate the efficacy of our method. Further analysis reveals the key factors easing cross-scale knowledge transfer and provides insights into the nature of latent semantic alignment.

翻译：大语言模型（LLMs）在其海量参数中编码了丰富的知识，这些知识可被定位、追踪和分析。尽管神经可解释性研究已取得进展，但如何以细粒度方式迁移知识，即参数化知识迁移（PKT），仍不明确。一个关键问题是如何在不同尺度的LLMs之间实现有效且高效的知识迁移，这对于在LLMs之间实现更灵活、更广泛的知识迁移应用至关重要。由于神经不兼容性——指不同尺度LLMs之间的架构和参数差异——现有直接复用层参数的方法受到严重限制。本文提出，潜在空间中的语义对齐是实现LLM跨尺度知识迁移的根本前提。我们的方法不直接使用层参数，而是以激活值作为逐层知识迁移的媒介。通过利用潜在空间中的语义信息，该方法简洁有效，优于先前工作，能更好地在不同尺度的模型间对齐行为。在四个基准测试上的评估证明了我们方法的有效性。进一步分析揭示了缓解跨尺度知识迁移的关键因素，并为潜在语义对齐的本质提供了见解。