Practical uses of Artificial Intelligence (AI) in the real world have demonstrated the importance of embedding moral choices into intelligent agents. They have also highlighted that defining top-down ethical constraints on AI according to any one type of morality is extremely challenging and can pose risks. A bottom-up learning approach may be more appropriate for studying and developing ethical behavior in AI agents. In particular, we believe that an interesting and insightful starting point is the analysis of emergent behavior of Reinforcement Learning (RL) agents that act according to a predefined set of moral rewards in social dilemmas. In this work, we present a systematic analysis of the choices made by intrinsically-motivated RL agents whose rewards are based on moral theories. We aim to design reward structures that are simplified yet representative of a set of key ethical systems. Therefore, we first define moral reward functions that distinguish between consequence- and norm-based agents, between morality based on societal norms or internal virtues, and between single- and mixed-virtue (e.g., multi-objective) methodologies. Then, we evaluate our approach by modeling repeated dyadic interactions between learning moral agents in three iterated social dilemma games (Prisoner's Dilemma, Volunteer's Dilemma and Stag Hunt). We analyze the impact of different types of morality on the emergence of cooperation, defection or exploitation, and the corresponding social outcomes. Finally, we discuss the implications of these findings for the development of moral agents in artificial and mixed human-AI societies.

翻译：人工智能在现实世界中的实际应用已表明，将道德选择嵌入智能体的重要性。同时，这些应用也指出，以自上而下的方式根据单一道德类型对人工智能施加伦理约束极为困难，且可能带来风险。自下而上的学习方法可能更适合研究和开发人工智能智能体的道德行为。特别是，我们认为一个有趣且富有洞察力的起点是分析在社交困境中依据预设道德奖励集行动、基于强化学习的智能体所涌现出的行为。在本研究中，我们系统分析了由内在动机驱动的强化学习智能体（其奖励基于道德理论）所作出的选择。我们旨在设计虽简化但仍具有代表性、涵盖一系列关键伦理体系的奖励结构。因此，我们首先定义了区分基于结果与基于规范的智能体、基于社会规范或内在美德的道德类型、以及基于单一美德与混合美德（如多目标方法论）的道德奖励函数。接着，我们通过在三个迭代式社会困境博弈（囚徒困境、志愿者困境和猎鹿博弈）中对学习型道德智能体之间的重复二元交互进行建模，来评估我们的方法。我们分析了不同道德类型对合作、背叛或剥削行为的涌现及其相应社会结果的影响。最后，我们讨论了这些发现对在人工社会及人机混合社会中发展道德智能体的启示。