We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the input text tokens at higher transformer layers. Then, a zero-init attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With efficient training, LLaMA-Adapter generates high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Furthermore, our approach can be simply extended to multi-modal input, e.g., images, for image-conditioned LLaMA, which achieves superior reasoning capacity on ScienceQA. We release our code at https://github.com/ZrrSkywalker/LLaMA-Adapter.

翻译：我们提出LLaMA-Adapter——一种轻量级适配方法，可高效地将LLaMA微调为指令跟随模型。采用52K条自指导演示数据后，LLaMA-Adapter仅需在冻结的LLaMA 7B模型上引入1.2M可学习参数，且可在8块A100 GPU上以不到一小时完成微调。具体而言，我们采用一组可学习的适配提示，将其拼接到高层Transformer层的输入文本令牌前。随后提出一种配备零门控机制的零初始化注意力机制，在自适应地将新指令线索注入LLaMA的同时，有效保留其预训练知识。通过高效训练，LLaMA-Adapter生成的高质量响应可与完全微调7B参数的Alpaca模型相媲美。此外，本方法可简单扩展至多模态输入（如图像），实现图像条件化LLaMA，在ScienceQA上展现出卓越的推理能力。相关代码已开源至https://github.com/ZrrSkywalker/LLaMA-Adapter。