When researchers iteratively refine ideas with large language models, do the models preserve fidelity to the original objective? We introduce DriftBench, a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation. Across 2,146 scored benchmark runs spanning seven models from five providers (including two open-weight), four interaction conditions, and 38 research briefs from 24 scientific domains, we find that iterative pressure reliably increases structural complexity and often reduces adherence to original constraints. A restatement probe reveals a dissociation between declarative recall and behavioral adherence, as models accurately restate constraints they simultaneously violate. The knows-but-violates (KBV) rate, measuring constraint non-compliance despite preserved recall, ranges from 8% to 99% across models. Structured checkpointing partially reduces KBV rates but does not close the dissociation, and complexity inflation persists. Human validation against blind raters confirms that the LLM judge under-detects constraint violations, making reported constraint adherence scores conservative. Sensitivity analyses confirm the findings are robust to temperature (0.7 vs.\ 1.0) and pressure type (novelty vs.\ rigor). We release all briefs, prompts, rubrics, transcripts, and scores as an open benchmark.
翻译:当研究人员与大语言模型迭代优化想法时,模型是否始终保持对原始目标的忠实?我们引入DriftBench,一个用于评估多轮LLM辅助科学构思中约束遵循的基准。通过对来自五个提供商(含两个开放权重模型)的七种模型、四种交互条件以及来自24个科学领域的38份研究简报进行的2146次评分基准运行,我们发现迭代压力可靠地增加结构复杂性,并且通常降低对原始约束的遵循。一项复述探测揭示了陈述性回忆与行为遵循之间的分离,因为模型能够准确复述其同时违反的约束。尽管保留了回忆,但衡量约束违规率的“知晓但违反”率(KBV)在各模型间范围为8%至99%。结构化检查点部分降低了KBV率,但并未消除这种分离,且复杂性膨胀持续存在。针对盲评员进行的人工验证确认,LLM裁判对约束违规的检测不足,因此报告的约束遵循分数是保守的。敏感性分析证实,这些发现对温度(0.7 vs. 1.0)和压力类型(新颖性 vs. 严谨性)具有鲁棒性。我们公开所有简报、提示、评分标准、记录和分数,作为开放基准。