Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment.

翻译：奖励分解通过将奖励分解为共享基函数和用户特定权重，实现大语言模型（LLM）的个性化。然而，现有方法孤立地从稀疏数据中估计用户权重并将其视为确定性点，导致推断不准确且不可靠。我们提出变分奖励分解（VRF）——一种不确定性感知框架，将每位用户的偏好表示为共享偏好空间中的变分分布。VRF通过变分编码器推断用户分布，通过Wasserstein距离匹配与共享概率基推导权重，并通过方差衰减损失降低不确定性估计的权重。在三个基准测试中，VRF在可见与未见用户、小样本场景及不同不确定性水平下均优于所有基线方法，其增益可持续至下游对齐任务。