RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in particular are now embedded not only in evaluation frameworks but also in the architectures of RAG systems themselves. While this integration can lead to genuine improvements, it also creates a risk of faulty measurements due to circularity. In this paper, we investigate this risk through comparative experiments with nugget-based RAG systems, including Ginger and Crucible, against strong baselines such as GPT-Researcher. By deliberately modifying Crucible to generate outputs optimized for an LLM judge, we show that near-perfect evaluation scores can be achieved when elements of the evaluation - such as prompt templates or gold nuggets - are leaked or can be predicted. Our results highlight the importance of blind evaluation settings and methodological diversity to guard against mistaking metric overfitting for genuine system progress.

翻译：RAG系统正越来越多地使用大语言模型裁判进行评估和优化，这一方法正迅速成为系统评估的主流范式。特别是基于摘要点的评估方法，目前不仅嵌入评估框架中，也融入了RAG系统本身的架构。虽然这种整合可能带来实质性改进，但也因循环性而引发错误测量的风险。本文通过针对基于摘要点的RAG系统（包括Ginger和Crucible）与GPT-Researcher等强基线系统的对比实验来研究这一风险。通过故意修改Crucible使其生成针对大语言模型裁判优化的输出，我们证明当评估要素（如提示模板或金摘要点）被泄露或可预测时，可获得近乎完美的评估分数。我们的研究结果凸显了盲评估设置与方法多样性的重要性，以防止将度量过拟合误认为系统实质性进展。