Training strong large language models (LLMs) requires high-quality supervision, which is often scarce. Recent work shows that paired preference data from weak-weaker model pairs (e.g., Qwen3 4B over 1.7B), despite the limited quality of individual responses, can provide an effective supervision signal through relative quality deltas, which we term a "weak" signal. This motivates a key research question: can multiple "weak" signals be constructively aggregated for improving strong models (e.g., Qwen3 8B)? To this end, we propose Preference Delta Aggregation (PDA), the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging. To improve the composition of independently trained adapters, we introduce Geometric Alignment Merging (GAM), which aligns their left and right singular bases through orthogonal Procrustes rotations and averages the aligned factors. Evaluations on knowledge reasoning and agentic search show that PDA-GAM exceeds the strongest individual delta, with further gains along the evaluated sequences of additional deltas. Using three reasoning deltas and four agentic deltas, PDA-GAM improves Qwen3-8B by 6.8 and 7.3 points on average, respectively, exceeding the strongest single-delta baseline by 2.1 and 4.3 points. These configurations achieve the highest average scores among the evaluated merging methods, with similar gains observed on a second student model. Further analysis suggests that these gains come from composing complementary capabilities encoded across distinct preference deltas. Code is available at https://github.com/AlbertQiSun/Preference-Delta-Aggregation.
翻译:暂无翻译