Large Language Models (LLMs) have made unprecedented breakthroughs, yet their increasing integration into everyday life might raise societal risks due to generated unethical content. Despite extensive study on specific issues like bias, the intrinsic values of LLMs remain largely unexplored from a moral philosophy perspective. This work delves into ethical values utilizing Moral Foundation Theory. Moving beyond conventional discriminative evaluations with poor reliability, we propose DeNEVIL, a novel prompt generation algorithm tailored to dynamically exploit LLMs' value vulnerabilities and elicit the violation of ethics in a generative manner, revealing their underlying value inclinations. On such a basis, we construct MoralPrompt, a high-quality dataset comprising 2,397 prompts covering 500+ value principles, and then benchmark the intrinsic values across a spectrum of LLMs. We discovered that most models are essentially misaligned, necessitating further ethical value alignment. In response, we develop VILMO, an in-context alignment method that substantially enhances the value compliance of LLM outputs by learning to generate appropriate value instructions, outperforming existing competitors. Our methods are suitable for black-box and open-source models, offering a promising initial step in studying the ethical values of LLMs.

翻译：大语言模型（LLMs）取得了前所未有的突破，但其日益融入日常生活可能因生成不道德内容而引发社会风险。尽管对偏见等具体问题已有广泛研究，但从道德哲学视角探索LLMs的内在价值仍未深入。本研究借助道德基础理论深入剖析伦理价值。为突破传统判别式评估可靠性差的局限，我们提出DeNEVIL——一种专为动态挖掘LLMs价值漏洞而设计的新型提示生成算法，能以生成式方式诱发违背伦理的行为，揭示其潜在的价值倾向。在此基础上，我们构建了包含2397条提示、覆盖500余条价值准则的高质量数据集MoralPrompt，并对多种LLMs的内在价值进行了基准测试。研究发现，大多数模型本质上存在价值观偏差，亟需进一步的伦理价值对齐。为此，我们开发了VILMO——一种情境内对齐方法，通过学习生成恰当的价值指令来显著提升LLM输出的价值合规性，性能超越现有竞争者。该方法适用于黑盒与开源模型，为研究LLMs伦理价值迈出了富有前景的第一步。