The adoption of generative AI across commercial and legal professions offers dramatic efficiency gains -- yet for law in particular, it introduces a perilous failure mode in which the AI fabricates fictitious case law, statutes, and judicial holdings that appear entirely authentic. Attorneys who unknowingly file such fabrications face professional sanctions, malpractice exposure, and reputational harm, while courts confront a novel threat to the integrity of the adversarial process. This failure mode is commonly dismissed as random `hallucination', but recent physics-based analysis of the Transformer's core mechanism reveals a deterministic component: the AI's internal state can cross a calculable threshold, causing its output to flip from reliable legal reasoning to authoritative-sounding fabrication. Here we present this science in a legal-industry setting, walking through a simulated brief-drafting scenario. Our analysis suggests that fabrication risk is not an anomalous glitch but a foreseeable consequence of the technology's design, with direct implications for the evolving duty of technological competence. We propose that legal professionals, courts, and regulators replace the outdated `black box' mental model with verification protocols based on how these systems actually fail.
翻译:生成式AI在商业和法律行业的应用带来了显著的效率提升——然而,尤其对法律领域而言,它引入了一种危险的失败模式:AI会编造出看似完全真实的虚构判例法、成文法和司法裁决。律师若在不知情的情况下提交此类虚构内容,将面临职业纪律处分、执业过失问责以及声誉损害,而法院则面临对抗制程序完整性的新型威胁。这种失败模式通常被归结为随机的"幻觉",但近期基于物理学对Transformer核心机制的分析揭示了一个确定性因素:AI的内部状态可能跨越可计算阈值,导致其输出从可靠的法律推理转变为看似权威的虚构陈述。本文在法律行业背景下阐释了这一科学原理,通过模拟案情摘要起草场景进行推演。我们的分析表明,虚构风险并非异常故障,而是技术设计可预见的后果,这对不断发展的技术胜任义务具有直接影响。我们建议法律从业者、法院和监管机构摒弃过时的"黑箱"思维模式,转而采用基于这些系统实际失效机制的验证协议。