Systems for making determinations on socially-constructed and complex concepts at scale are increasingly being deployed. To make such fuzzy concepts tractable for training and evaluating AI, aligning model outputs, or human-in-the-loop workflows, the prevailing strategy involves developing `constitutions' in the form of rules, policies, or principles. However, high-level rules often fail to capture situational nuances or have differing interpretations, resulting in inconsistent decisions. In this work, we introduce case law grounding (CLG), a hybrid workflow inspired by case law in the legal realm where past judgments on specific cases inform new decisions. Evaluating on two task domains, we find that CLG can improve alignment of decisions (+9.6% and +10.9% accuracy) and consistency ($\Delta\bar{\kappa}$ of +0.263 and +0.433) of human decision-makers, while also providing auditable rationales. We also find similarly substantial alignment improvements for an LLM decision-maker (+25% and +23% accuracy).

翻译：针对大规模部署的涉及社会建构复杂概念的系统，为使此类模糊概念在AI训练与评估、模型输出对齐或人机协作工作流中具有可操作性，现行策略主要采用规则、政策或原则等形式的"宪章"。然而，高层级规则常无法捕捉情境差异或导致解读分歧，造成决策不一致。本研究提出判例法 grounding（CLG）——一种受法律领域判例法（即既往特定案例裁判指引新决策）启发的混合工作流。在两个任务领域评估发现，CLG可提升人类决策者的对齐度（准确率+9.6%和+10.9%）及一致性（$\Delta\bar{\kappa}$：+0.263和+0.433），同时提供可审计的决策理由。对LLM决策者而言，CLG同样带来显著的对齐改善（准确率+25%和+23%）。