Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA-1, Vicuna-v1.3, Mistral-7b, and Mistral-7b-Instruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.

翻译：大型语言模型首先通过数万亿token进行预训练，随后通过指令微调或对齐以满足特定偏好。由于预训练所需计算资源过大，大多数研究者难以企及，但借助LoRA和QLoRA等参数高效方法，微调已变得切实可行。对齐过程对诸多因素敏感，包括数据数量与质量、对齐方法及适配器秩（adapter rank）。然而，目前尚缺乏对这些因素如何影响下游性能的系统性研究。为填补这一空白，我们针对三个关键维度中的常见选择展开深度探究：（i）对齐数据集（HH-RLHF和BeaverTails）、（ii）对齐技术（SFT和DPO）及（iii）模型（LLaMA-1、Vicuna-v1.3、Mistral-7b和Mistral-7b-Instruct）。涵盖300余项实验的广泛配置揭示了若干一致性趋势与意外发现：信息更丰富的数据有助于偏好对齐；监督式微调在某些场景下优于偏好优化；针对特定偏好的对齐能提升下游任务性能。通过深度分析，我们提出关键性指导原则，以帮助研究者更有效地实现参数高效的大语言模型对齐。