Large language models systematically hallucinate legal citations -- fabricating statute references, citing repealed provisions, and confusing jurisdictions -- yet no automated method exists to measure or reduce this behavior at scale. We propose citation grounding (CG), a metric that verifies LLM-generated legal citations against a ground-truth citation graph extracted from 100.8 million Ukrainian court decisions (502 million edges, 21,736 unique statute nodes). CG decomposes into three components -- citation precision (does the cited provision exist?), citation relevance (is it contextually appropriate?), and citation temporality (was it valid at the relevant date?) -- enabling differential diagnosis of hallucination types. Empirical evaluation on 100 Ukrainian legal queries across five systems -- four commercial LLMs via AWS Bedrock (Claude Haiku 4.5, Mistral Pixtral Large, Amazon Nova Pro/Lite) and one RAG-augmented production system -- reveals CG ranging from 0.791 to 0.873, with 13-21% of citations hallucinated. To reduce hallucinations without human annotation, we introduce Citation Grounding DPO (CG-DPO): a method that constructs preference pairs algorithmically by corrupting verified citations from real court decisions via four targeted strategies. On a dataset of 2,244 court decisions, a Qwen2.5-7B-Instruct model fine-tuned with LoRA achieves 98.5% mean validation accuracy in distinguishing correct from corrupted citations (rewards margin +14.9, std < 0.3 pp across 3 seeds). The citation graph, evaluation framework, and CG-DPO dataset are released as open resources.
翻译:摘要:大语言模型会系统性产生法律引用幻觉——捏造法规引用、引用已废止条款、混淆司法管辖区——但目前尚无自动化方法可在规模化层面度量或减少该行为。我们提出“引用溯源”(CG)度量指标,通过将大语言模型生成的法律引用与从1.008亿份乌克兰法院判决中提取的基准引用图(含5.02亿条边、21,736个独立法规节点)进行核验。CG分解为三个子指标——引用精确性(所引条款是否存在)、引用相关性(是否契合上下文)及引用时效性(在相关日期是否有效),从而实现对幻觉类型的差异化诊断。基于五个系统对100个乌克兰法律查询进行的实证评估——包括通过AWS Bedrock部署的四个商业大语言模型(Claude Haiku 4.5、Mistral Pixtral Large、Amazon Nova Pro/Lite)及一个经RAG增强的生产系统——显示CG值介于0.791至0.873之间,其中13-21%的引用存在幻觉。为在无需人工标注条件下减少幻觉,我们提出“引用溯源强化学习”(CG-DPO):该方法通过四种定向策略对真实法院判决中的已验证引用进行扰动,算法化构建偏好配对。基于包含2,244份法院判决的数据集,采用LoRA微调的Qwen2.5-7B-Instruct模型在区分正确与扰动引用时达到98.5%的平均验证准确率(奖励边际+14.9,三组随机种子标准差<0.3个百分点)。该引用图、评估框架及CG-DPO数据集已作为开放资源发布。