Value-aware AI should recognise human values and adapt to the value systems (value-based preferences) of different users. This requires operationalization of values, which can be prone to misspecification. The social nature of values demands their representation to adhere to multiple users while value systems are diverse, yet exhibit patterns among groups. In sequential decision making, efforts have been made towards personalization for different goals or values from demonstrations of diverse agents. However, these approaches demand manually designed features or lack value-based interpretability and/or adaptability to diverse user preferences. We propose algorithms for learning models of value alignment and value systems for a society of agents in Markov Decision Processes (MDPs), based on clustering and preference-based multi-objective reinforcement learning (PbMORL). We jointly learn socially-derived value alignment models (groundings) and a set of value systems that concisely represent different groups of users (clusters) in a society. Each cluster consists of a value system representing the value-based preferences of its members and an approximately Pareto-optimal policy that reflects behaviours aligned with this value system. We evaluate our method against a state-of-the-art PbMORL algorithm and baselines on two MDPs with human values.

翻译：价值感知人工智能应能识别人类价值观并适应不同用户的价值系统（基于价值的偏好）。这需要对价值观进行操作化处理，而该过程容易产生设定偏差。价值观的社会属性要求其表征需同时满足多用户需求，而价值系统虽具有多样性，却能在群体间呈现规律性模式。在序列决策领域，已有研究致力于通过分析不同智能体的行为示范来实现针对多元目标或价值观的个性化适配。然而，现有方法或依赖人工设计特征，或缺乏基于价值的可解释性及对多样化用户偏好的适应能力。本文提出基于聚类与偏好多目标强化学习（PbMORL）的算法，用于在马尔可夫决策过程（MDPs）中学习智能体社会的价值对齐模型与价值系统。我们通过联合学习社会衍生的价值对齐模型（基础表征）以及一组能简洁表征社会中不同用户群体（聚类）的价值系统。每个聚类包含代表其成员价值偏好的价值系统，以及反映符合该价值系统行为的近似帕累托最优策略。我们在两个包含人类价值观的MDP环境中，将所提方法与最先进的PbMORL算法及基线模型进行了对比评估。