Current frameworks for consensus statement generation with large language models lack the inherent structure needed to provide provable fairness guarantees when aggregating diverse free-form opinions. We model the task as a multi-objective, token-level Markov Decision Process (MDP), where each objective corresponds to an agent's preference. Token-level rewards for each agent are derived from their policy (e.g., a personalized language model). This approach utilizes the finding that such policies implicitly define optimal Q-functions, providing a principled way to quantify rewards at each generation step without a value function (Rafailov et al., 2024). This MDP formulation creates a formal structure amenable to analysis using principles from social choice theory. We propose two approaches grounded in social choice theory. First, we propose a stochastic generation policy guaranteed to be in the ex-ante core, extending core stability concepts from voting theory to text generation. This policy is derived from an underlying distribution over complete statements that maximizes proportional fairness (Nash Welfare). Second, for generating a single statement, we target the maximization of egalitarian welfare using search algorithms within the MDP framework. Empirically, experiments using language models to instantiate agent policies show that search guided by the egalitarian objective generates consensus statements with improved worst-case agent alignment compared to baseline methods, including the Habermas Machine (Tessler et al., 2024).

翻译：当前基于大语言模型的共识声明生成框架在聚合多样化自由形式意见时，缺乏提供可证明公平性保证的内在结构。我们将该任务建模为多目标令牌级马尔可夫决策过程，其中每个目标对应一个智能体的偏好。每个智能体的令牌级奖励源自其策略（例如个性化语言模型）。该方法利用了此类策略隐式定义最优Q函数的发现，从而提供了一种无需价值函数即可量化每个生成步骤奖励的原则性方法（Rafailov等人，2024）。这种MDP表述创建了一个适用于社会选择理论原则分析的形式化结构。我们提出了两种基于社会选择理论的方法。首先，我们提出了一种随机生成策略，该策略保证处于事前核心，将投票理论中的核心稳定性概念扩展到文本生成领域。该策略源自一个最大化比例公平性的完整声明基础分布。其次，对于生成单个声明，我们通过在MDP框架内使用搜索算法，以最大化平等主义福利为目标。实证研究表明，使用语言模型实例化智能体策略时，与包括哈贝马斯机器在内的基线方法相比，以平等主义目标为指导的搜索生成的共识声明在改善最坏情况智能体对齐方面表现更优。