We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/Yewei-Liu/SHINE

翻译：我们提出SHINE（可扩展上下文超网络），这是一种可扩展的超网络，能够将多样化的有意义的上下文映射为大型语言模型（LLM）的高质量LoRA适配器。通过复用冻结LLM自身的参数进行上下文超网络设计，并引入架构创新，SHINE克服了先前超网络的关键限制，以相对较少的参数实现了强大的表达能力。我们引入了预训练与指令微调流程，训练该超网络以单次前向传播从多样化的有意义的上下文中生成高质量LoRA适配器。该方法无需任何微调即可更新LLM参数，并能立即执行与上下文相关的复杂问答任务而无需直接访问上下文，从而在单次传播中有效地将上下文知识转化为参数内知识。我们的工作在多种任务上取得了优异的结果，与基于SFT的LLM适配方法相比，极大地节省了时间、计算与内存成本，并展现出巨大的扩展潜力。代码已开源：https://github.com/Yewei-Liu/SHINE