Recently, the instruction-tuning of large language models is a crucial area of research in the field of natural language processing. Due to resource and cost limitations, several researchers have employed parameter-efficient tuning techniques, such as LoRA, for instruction tuning, and have obtained encouraging results In comparison to full-parameter fine-tuning, LoRA-based tuning demonstrates salient benefits in terms of training costs. In this study, we undertook experimental comparisons between full-parameter fine-tuning and LoRA-based tuning methods, utilizing LLaMA as the base model. The experimental results show that the selection of the foundational model, training dataset scale, learnable parameter quantity, and model training cost are all important factors. We hope that the experimental conclusions of this paper can provide inspiration for training large language models, especially in the field of Chinese, and help researchers find a better trade-off strategy between training cost and model performance. To facilitate the reproduction of the paper's results, the dataset, model and code will be released.

翻译：近期，大语言模型的指令调优是自然语言处理领域的关键研究课题。受资源与成本限制，部分研究者采用参数高效调优技术（如LoRA）进行指令调优，并取得了令人鼓舞的成果。相较于全参数微调，基于LoRA的调优方法在训练成本方面展现出显著优势。本研究以LLaMA为基础模型，对全参数微调与基于LoRA的调优方法进行了实验对比。实验结果表明，基础模型选择、训练数据集规模、可学习参数量及模型训练成本均为重要影响因素。期望本文的实验结论能为大语言模型训练（尤其是中文领域）提供启示，帮助研究者探索训练成本与模型性能之间的更优平衡策略。为便于复现本文结果，相关数据集、模型及代码将予以公开。