Making moral judgments is an essential step toward developing ethical AI systems. Prevalent approaches are mostly implemented in a bottom-up manner, which uses a large set of annotated data to train models based on crowd-sourced opinions about morality. These approaches have been criticized for potentially overgeneralizing a limited group of annotators' moral stances and lacking explainability. In contrast, top-down approaches make moral judgments grounded in a set of principles. However, it remains conceptual due to the incapability of previous language models and the unsolved debate among moral principles. In this study, we propose a flexible framework to steer Large Language Models (LLMs) to perform moral reasoning with well-established moral theories from interdisciplinary research. The theory-guided top-down framework can incorporate various moral theories. Our experiments demonstrate the effectiveness of the proposed framework on datasets derived from moral theories. Furthermore, we show the alignment between different moral theories and existing morality datasets. Our analysis exhibits the potentials and flaws in existing resources (models and datasets) in developing explainable moral judgment-making systems.

翻译：做出道德判断是开发伦理人工智能系统的关键一步。当前主流方法大多采用自下而上的方式，利用大规模标注数据，基于众包道德意见训练模型。这些方法因可能过度泛化有限标注者的道德立场且缺乏可解释性而受到批评。相比之下，自上而下的方法基于一套原则进行道德判断。然而，由于以往语言模型的能力不足以及道德原则间悬而未决的争论，该方法一直停留在概念层面。在本研究中，我们提出了一种灵活的框架，引导大语言模型依据跨学科研究中成熟的道德理论进行道德推理。这种理论指导的自上而下框架能够整合多种道德理论。我们的实验证明了该框架在基于道德理论的数据集上的有效性。此外，我们展示了不同道德理论与现有道德数据集之间的对应关系。我们的分析揭示了现有资源（模型和数据集）在开发可解释的道德判断系统方面的潜力与缺陷。