The Deliberative Reason Index (DRI) is increasingly used to assess the coherence between considerations and preferences in deliberative settings, including applications to LLM-generated data. Under low-signal conditions, however, the standard DRI can produce inflated scores by treating near-zero correlations as evidence of consistency. Monte Carlo simulations across common study designs show that this bias increases with group size and yields positive values even under random response. A modified DRI is introduced that applies a continuous penalty to low-signal correlation pairs. The modification preserves the original scale and reduces exactly to the standard DRI when substantive signal is present. A threshold sensitivity analysis identifies τ=0.2as the optimal parameter. An empirical check with archival deliberative data shows that substantive inferences remain unchanged. The modification improves the reliability and comparability of the DRI in low-signal settings.
翻译:审议理由指数(DRI)日益用于评估审议环境中考量与偏好之间的一致性,包括应用于大型语言模型生成的数据。然而,在低信号条件下,标准DRI可能通过将近零相关性视为一致性证据,从而产生膨胀的分数。跨常见研究设计的蒙特卡洛模拟显示,这种偏差随群体规模增大而加剧,即使在随机响应下也会产生正值。本文引入一种修正的DRI,其对低信号相关对施加连续惩罚。该修正保留了原始量表,并在存在实质性信号时精确还原为标准DRI。阈值敏感性分析确定τ=0.2为最优参数。对存档审议数据的实证检验表明,实质性推论保持不变。该修正提升了低信号设置下DRI的可靠性与可比性。