Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.

翻译：联邦学习为大语言模型的对齐提供了一条隐私保护路径，但现有框架通常强制采用统一奖励模型，不可避免地平均了本质冲突的用户偏好（如帮助性与无害性的权衡）。尽管变分偏好学习为个性化提供了途径，但将其适配至去中心化场景存在根本性挑战：由严重局部数据稀缺与异质性驱动的后验坍塌问题。本文提出融合Gumbel-Softmax先验的联邦变分偏好对齐框架，该框架旨在不损害隐私的前提下解耦多样偏好。为稳定变分推断，我们引入联邦混合先验，使客户端能够将聚合的总体分布作为动态先验加以利用。此外，我们整合了正交损失函数，在隐空间中显式强制偏好原型分离。在HH-RLHF数据集上的实验表明，所提框架显著优于统一基线模型，成功解耦了冲突的用户意图，并实现了动态偏好切换。