We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy. SimpleQA Verified was created through a rigorous multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation to produce a more reliable and challenging evaluation set, alongside improvements in the autorater prompt. On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5. This work provides the research community with a higher-fidelity tool to track genuine progress in parametric model factuality and to mitigate hallucinations. The benchmark dataset, evaluation code, and leaderboard are available at: https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified.

翻译：我们提出了SimpleQA Verified，这是一个基于OpenAI SimpleQA构建的、包含1000个提示的基准测试集，用于评估大语言模型在短文本形式上的事实性。该基准解决了OpenAI原基准中存在的关键局限性，包括标签噪声与错误、主题偏见以及问题冗余。SimpleQA Verified通过一个严谨的多阶段过滤流程创建，该流程涉及去重、主题平衡和来源核对，从而生成了一个更可靠且更具挑战性的评估集，同时对自动评分提示进行了改进。在这一新基准上，Gemini 2.5 Pro取得了55.6的最先进F1分数，表现优于包括GPT-5在内的其他前沿模型。这项工作为研究社区提供了一个更高保真度的工具，以追踪参数化模型在事实性方面的真实进展，并减轻幻觉问题。基准数据集、评估代码和排行榜可在以下网址获取：https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified。