AI systems deployed in legal workflows hallucinate at rates that aggregate metrics report at ~52%, but this average conceals where errors concentrate and in which direction they run, leaving compliance officers without an actionable signal for trustworthy deployment. We present LegalHalluLens, an auditing framework with three components: typed hallucination profiles across four legally-motivated claim categories (numeric, temporal, obligation/entitlement, factual) over CUAD (Hendrycks et al., 2021); a Risk Direction Index (RDI) that reduces omission-versus-invention bias to a single deployment-comparable scalar; and a typed debate pipeline calibrated to both magnitudes and directions. Across 510 contracts and 249,252 clause-level instances we measure a within-model gap of approximately 38-40 pp between obligation/numeric and temporal claims that aggregate reporting hides, and show that two systems with matched 52% rates can carry opposite RDIs. The debate pipeline reduces fabricated detections by 45% with per-category gains tracking the diagnosis, matching commercial APIs with a substantially smaller backbone (4B active parameters). Typed profiles and RDI surface failure modes that aggregate metrics hide; we further show these diagnostics serve as calibration inputs for multi-agent debate pipelines, where Skeptic challenges and asymmetric gates targeted at measured failure modes outperform generically-tuned debate. The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.

翻译：摘要：部署于法律工作流程中的AI系统，其总括指标报告的幻觉率约为52%，但这一均值掩盖了错误集中区域及其偏差方向，导致合规官员缺乏可信部署所需的行为信号。我们提出LegalHalluLens审计框架，包含三个组件：基于CUAD数据集（Hendrycks et al., 2021）构建的、涵盖四类法律驱动声明（数值型、时间型、义务/权利型、事实型）的类型化幻觉画像；风险方向指数（RDI），将遗漏偏差与捏造偏差缩减为单一可跨部署比较的标量；以及经幅度与方向校准的类型化辩论流水线。对510份合同与249,252个条款级实例的测量表明，模型内数值/义务类声明与时间类声明之间存在约38-40个百分点的差距（总括报告所隐藏），并揭示两个具有相同52%幻觉率的系统可能呈现相反的RDI。该辩论流水线将伪造检测减少45%，各类别改进幅度与诊断结果一致，且以大幅精简的骨干网络（40亿活跃参数）达到与商业API相当的性能。类型化画像与RDI揭示了总括指标所掩盖的故障模式；我们进一步证明这些诊断结果可作为多智能体辩论流水线的校准输入——其中，针对已测量故障模式的质疑者挑战与非对称门控机制优于通用调优式辩论。该框架支持面向法律AI在真实环境中部署的方向感知采购、问责制与智能体设计。