A Large Language Model (LLM) is considered consistent if semantically equivalent prompts produce semantically equivalent responses. Despite recent advancements showcasing the impressive capabilities of LLMs in conversational systems, we show that even state-of-the-art LLMs are highly inconsistent in their generations, questioning their reliability. Prior research has tried to measure this with task-specific accuracy. However, this approach is unsuitable for moral scenarios, such as the trolley problem, with no "correct" answer. To address this issue, we propose a novel information-theoretic measure called Semantic Graph Entropy (SGE) to measure the consistency of an LLM in moral scenarios. We leverage "Rules of Thumb" (RoTs) to explain a model's decision-making strategies and further enhance our metric. Compared to existing consistency metrics, SGE correlates better with human judgments across five LLMs. In the future, we aim to investigate the root causes of LLM inconsistencies and propose improvements.

翻译：大型语言模型（LLM）若能在语义等价的提示下生成语义等价的响应，则被认为具有一致性。尽管近期研究展示了LLM在对话系统中令人瞩目的能力，但我们发现，即便是最先进的LLM在生成过程中也存在高度不一致性，这对其可靠性提出了质疑。以往研究尝试通过特定任务的精确度来衡量这一点，然而，这种方法并不适用于道德场景，例如电车难题，因为这类问题不存在“正确”答案。为解决这一问题，我们提出了一种名为语义图熵（Semantic Graph Entropy，简称SGE）的创新型信息论度量指标，用于衡量LLM在道德场景中的一致性。我们利用“经验法则”（Rules of Thumb，简称RoTs）来解释模型的决策策略，并进一步优化了我们的度量指标。与现有的一致性度量相比，SGE在五个LLM上更能与人类判断保持高度一致。未来，我们旨在探究LLM不一致性的根本原因，并提出改进方案。