Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, both of these approaches face key limitations: ICL is inefficient when handling many demonstrations, and SFT incurs training overhead while sacrificing flexibility. Mapping instructions or demonstrations from context directly into adapter parameters offers an appealing alternative. While prior work explored generating adapters based on a single input context, it has overlooked the need to integrate multiple chunks of information. To address this gap, we introduce CompAs, a meta-learning framework that translates context into adapter parameters with a compositional structure. Adapters generated this way can be merged algebraically, enabling instructions, demonstrations, or retrieved passages to be seamlessly combined without reprocessing long prompts. Critically, this approach yields three benefits: lower inference cost, robustness to long-context instability, and establishes a principled solution when input exceeds the model's context window. Furthermore, CompAs encodes information into adapter parameters in a reversible manner, enabling recovery of input context through a decoder, facilitating safety and security. Empirical results on diverse multiple-choice and extractive question answering tasks show that CompAs outperforms ICL and prior generator-based methods, especially when scaling to more inputs. Our work establishes composable adapter generation as a practical and efficient alternative for scaling LLM deployment.

翻译：大型语言模型（LLM）通常能够通过上下文学习（ICL）或有监督微调（SFT）无缝适应新任务。然而，这两种方法均面临关键局限：ICL在处理大量示例时效率低下，而SFT则会产生训练开销并牺牲灵活性。将上下文中的指令或示例直接映射到适配器参数提供了一种有吸引力的替代方案。尽管先前的研究探索了基于单一输入上下文生成适配器的方法，但其忽视了整合多个信息片段的需求。为填补这一空白，我们提出了CompAs——一种将上下文转换为具有组合结构的适配器参数的元学习框架。通过这种方式生成的适配器可以进行代数合并，使得指令、示例或检索到的段落能够无缝结合，而无需重新处理长提示。重要的是，该方法带来三大优势：更低的推理成本、对长上下文不稳定的鲁棒性，以及当输入超出模型上下文窗口时提供一种原则性解决方案。此外，CompAs以可逆的方式将信息编码到适配器参数中，使得能够通过解码器恢复输入上下文，从而增强安全性与可靠性。在多样化的多项选择与抽取式问答任务上的实验结果表明，CompAs在扩展到更多输入时，其性能优于ICL及先前基于生成器的方法。我们的工作确立了可组合适配器生成作为一种实用且高效的替代方案，可用于扩展LLM的部署。