While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.

翻译：尽管大语言模型常通过微调来解锁其在下游应用中的能力，但我们对于不同微调方法的归纳偏差（特别是其缩放特性）仍理解有限。为填补这一空白，我们开展了系统性实验，研究不同缩放因子——包括大语言模型规模、预训练数据规模、新增微调参数量及微调数据规模——是否及如何影响微调性能。我们考虑两类微调方法：全模型微调与参数高效微调（包括提示微调和LoRA），并在大语言模型规模显著大于微调数据规模的数据受限场景下探究其缩放行为。基于两组从1B到16B参数的预训练双语大语言模型，以及在双语机器翻译和多语言摘要基准上的实验，我们发现：1）大语言模型微调遵循微调数据规模与每个其他缩放因子之间的基于乘数幂律的联合缩放定律；2）大语言模型微调从模型规模缩放中获益多于预训练数据缩放，且参数高效微调的参数缩放通常无效；3）最优微调方法高度依赖于任务和微调数据。我们希望这些发现能为理解、选择和发展大语言模型微调方法提供启示。