Gradient-based preference optimization methods for large language model (LLM) alignment suffer from preference collapse, converging to narrow behavioral modes while neglecting preference diversity. We introduce EvoPref, a multi-objective evolutionary algorithm that maintains populations of Low-Rank Adaptation (LoRA) adapters optimized across helpfulness, harmlessness, and honesty objectives using Non-dominated Sorting Genetic Algorithm II (NSGA-II) selection with archive-based diversity preservation. Our primary contribution is demonstrating that population-based methods discover substantially more diverse alignments than gradient descent. On standard benchmarks, EvoPref improves preference coverage by 18% (median 82.5% vs. 70.0% for ORPO, $p<0.001$, Wilcoxon, $n=30$) and reduces collapse rates by 47% (11.0% vs. 20.6%, $p<0.001$), while achieving competitive alignment quality (median 75.5% RewardBench vs. 75.0% for ORPO, $p<0.05$). We provide theoretical motivation extending recent multi-objective evolutionary algorithm (MOEA) runtime analysis (Dang et al., 2025) suggesting why archive-based methods escape collapse more effectively than single-trajectory optimization. Comprehensive comparisons against MOEA/D, SMS-EMOA, CMA-ES, and gradient baselines (DPO, IPO, KTO, ORPO) with rigorous statistical testing (Friedman with Holm correction, Vargha-Delaney effect sizes, median with IQR) confirm that multi-objective selection with diversity preservation is essential. This work establishes evolutionary optimization as a principled paradigm for diverse LLM alignment.

翻译：基于梯度的偏好优化方法在大语言模型对齐中面临偏好坍缩问题，即收敛至狭窄的行为模式而忽略偏好多样性。我们提出EvoPref——一种多目标进化算法，该算法维护低秩适配器种群，通过采用带存档多样性保持的非支配排序遗传算法II选择机制，以有用性、无害性和诚实性为目标进行优化。我们的主要贡献在于证明：基于种群的方法比梯度下降能发现显著更多样化的对齐模式。在标准基准测试中，EvoPref将偏好覆盖率提升18%（中位数82.5%对比ORPO的70.0%，Wilcoxon检验$p<0.001$，$n=30$），将坍缩率降低47%（11.0%对比20.6%，$p<0.001$），同时保持具有竞争力的对齐质量（RewardBench中位数75.5%对比ORPO的75.0%，$p<0.05$）。我们提供理论支撑，将近期多目标进化算法运行时分析（Dang等，2025）进行扩展，阐明为何基于存档的方法比单轨迹优化更能有效避免坍缩。通过与MOEA/D、SMS-EMOA、CMA-ES及梯度基线方法（DPO、IPO、KTO、ORPO）的全面比较（采用Friedman检验伴随Holm校正、Vargha-Delaney效应量、中位数与四分位距的严格统计检验）证实：具有多样性保持的多目标选择是必要条件。本工作确立了进化优化作为多样化大模型对齐的基本原则范式。