Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there are two different ways of allocating considered: more diverse PROMPTS or more diverse RESPONSES to be labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence. We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and conduct experiments to show its effect on different algorithms.

翻译：人类偏好对齐能够防止大型语言模型生成误导性或有害内容，但需要高成本的人工反馈。假设人工标注资源有限，存在两种不同的分配方式：标注更多样化的提示或更多样化的响应。然而，目前缺乏对两者影响的直接比较。在本研究中，我们首先根据微调样本数量控制双方的多样性，以直接反映其影响。我们发现，与大量提示相比，更多响应但更少提示能更好地触发语言模型实现人类对齐。此外，提示多样性的概念比通常以单一数字量化的响应多样性更为复杂。因此，我们提出了一种新的提示多样性公式，进一步揭示了它与微调后语言模型最终性能的线性相关性。我们还将其应用于数据增强，并通过实验展示了它对不同算法的影响。