The holy grail of LLM personalization is a single LLM for each user, perfectly aligned with that user's preferences. However, maintaining a separate LLM per user is impractical due to constraints on compute, memory, and system complexity. We address this challenge by developing a principled method for selecting a small portfolio of LLMs that captures representative behaviors across heterogeneous users. We model user preferences across multiple traits (e.g., safety, humor, brevity) through a multi-dimensional weight vector. Given reward functions across these dimensions, our algorithm PALM (Portfolio of Aligned LLMs) generates a small portfolio of LLMs such that, for any weight vector, the portfolio contains a near-optimal LLM for the corresponding scalarized objective. To the best of our knowledge, this is the first result that provides theoretical guarantees on both the size and approximation quality of LLM portfolios for personalization. It characterizes the trade-off between system cost and personalization, as well as the diversity of LLMs required to cover the landscape of user preferences. We provide empirical results that validate these guarantees and demonstrate greater output diversity over common baselines.

翻译：大型语言模型（LLM）个性化的理想目标是实现每位用户拥有一个独立的 LLM，并完美契合该用户的偏好。然而，由于计算、内存和系统复杂性的限制，为每位用户维护一个独立的 LLM 并不可行。我们通过开发一种原则性方法来解决这一挑战，该方法旨在选择一小部分 LLM，从而捕捉异构用户群体的代表性行为。我们通过多维权重向量对用户在多个特质（例如安全性、幽默感、简洁性）上的偏好进行建模。给定这些维度的奖励函数，我们的算法 PALM（对齐 LLM 组合）能够生成一个小的 LLM 组合，使得对于任意权重向量，该组合中都包含一个对应于相应标量化目标的近似最优 LLM。据我们所知，这是首个为 LLM 组合的规模及其在个性化中的近似质量提供理论保证的研究成果。它刻画了系统成本与个性化之间的权衡关系，以及覆盖用户偏好景观所需的 LLM 多样性。我们提供的实证结果验证了这些保证，并展示了相较于常见基线的更高输出多样性。