Reference labels for machine-learning benchmarks are increasingly synthesized with LLM assistance, but their reliability remains underexamined. We audit MedCalc-Bench, a clinical benchmark for medical score computation whose labels were partly derived with LLM assistance, and develop a scalable physician-in-the-loop stewardship pipeline to reassess them. At least 27% of test labels are likely erroneous or incomputable. On a 50-instance subset validated by physicians, our recomputed labels agree with physician ground truth 74% of the time (95% CI, 60-84%) versus 20% for the originals (95% CI, 11-33%). Using original labels to evaluate frontier LLMs underestimates accuracy by 16-23 percentage points. In a controlled reinforcement-learning experiment, a model trained on recomputed labels outperforms one trained on originals by 13.5 percentage points (95% CI, 10.6-16.6%) on physician-labeled instances, and this advantage extends to related medical tasks. LLM-assisted benchmarks can propagate systematic errors into both evaluation and post-training unless actively stewarded.

翻译：机器学习基准的参考标签越来越多地借助大语言模型（LLM）进行合成，但其可靠性仍缺乏系统验证。我们审计了MedCalc-Bench这一医学评分计算临床基准（其标签部分由LLM辅助生成），并开发了可扩展的医师参与式管理流水线以重新评估这些标签。至少27%的测试标签可能存在错误或无法计算。在经医师验证的50个实例子集中，我们重新计算的标签与医师金标准的一致性为74%（95%置信区间：60-84%），而原始标签仅为20%（95%置信区间：11-33%）。使用原始标签评估前沿LLM会低估其准确率16-23个百分点。在受控强化学习实验中，基于重新计算标签训练的模型在医师标注实例上的表现比基于原始标签训练的模型高出13.5个百分点（95%置信区间：10.6-16.6%），且该优势可延伸至相关医学任务。LLM辅助基准若缺乏主动管理，会将系统性错误传播至评估与后续训练阶段。