We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE

翻译：我们提出了SHINE（可扩展超上下文内网络），这是一种可扩展的超网络，能够将多样化的有意义上下文映射为高质量的大语言模型（LLM）LoRA适配器。通过利用冻结LLM自身的参数设计上下文内超网络架构，并引入架构创新，SHINE克服了以往超网络的关键局限，以相对较少的参数实现了强大的表达能力。我们引入了预训练和指令微调流程，训练超网络在单次前向传播中从多样化的有意义上下文生成高质量LoRA适配器。它无需任何微调即可更新LLM参数，并立即实现与上下文相关的复杂问答任务，同时无需直接访问上下文，从而有效地将上下文知识一次性转化为参数内知识。我们的工作在多种任务上取得了出色成果，相比基于SFT的LLM适应方法大幅节省了时间、计算和内存成本，并展现出强大的扩展潜力。我们的代码开源于https://github.com/MuLabPKU/SHINE