Vision-language alignment in multi-modal large language models (MLLMs) relies on supervised fine-tuning (SFT) or reinforcement learning (RL). To align multi-modal large language models (MLLMs) in the post-training stage, supervised fine-tuning (SFT) is a stable choice but requires human annotations and lacks task generalizations, while Reinforcement Learning (RL) searches for better answers from reward signals but suffers from computational overhead and instability. To achieve balance among scalability, efficiency, and alignment generalizations, we propose MergeMix, a unified paradigm that bridges SFT and RL with an efficient Token Merge based Mixup augmentation. As for the Mixup policy, we generate contextual aligned mixed images with the corresponding labels according to the merged attention maps with cluster regions. Then, we enhance the preference-driven paradigm for MLLMs by building preference pairs with raw images and MergeMix-generated ones and optimizing the soft preference margin with the mixed SimPO loss. Extensive experiments demonstrate that MergeMix not only achieves dominant classification accuracy as an augmentation method but also improves generalization abilities and alignment of MLLMs, providing a new learning paradigm for preference alignment with training efficiency and stability.

翻译：多模态大语言模型（MLLMs）中的视觉-语言对齐通常依赖于监督微调（SFT）或强化学习（RL）。为了在训练后阶段对齐多模态大语言模型（MLLMs），监督微调（SFT）是一种稳定的选择，但需要人工标注且缺乏任务泛化能力；而强化学习（RL）通过奖励信号搜索更优答案，却存在计算开销大和不稳定的问题。为了在可扩展性、效率和对齐泛化能力之间取得平衡，我们提出了MergeMix，这是一种统一范式，通过高效的基于Token Merge的Mixup增强来桥接SFT与RL。在Mixup策略方面，我们根据具有聚类区域的合并注意力图，生成上下文对齐的混合图像及其对应标签。随后，我们通过构建原始图像与MergeMix生成图像之间的偏好对，并利用混合SimPO损失优化软偏好边界，从而增强MLLMs的偏好驱动范式。大量实验表明，MergeMix不仅作为一种增强方法取得了主导性的分类准确率，还提升了MLLMs的泛化能力与对齐效果，为偏好对齐提供了一种兼具训练效率与稳定性的新学习范式。