This paper presents FormationEval, an open multiple-choice question benchmark for evaluating language models on petroleum geoscience and subsurface disciplines. The dataset contains 505 questions across seven domains including petrophysics, petroleum geology and reservoir engineering, derived from three authoritative sources using a reasoning model with detailed instructions and a concept-based approach that avoids verbatim copying of copyrighted text. Each question includes source metadata to support traceability and audit. The evaluation covers 72 models from major providers including OpenAI, Anthropic, Google, Meta and open-weight alternatives. The top performers achieve over 97\% accuracy, with Gemini 3 Pro Preview reaching 99.8\%, while tier and domain gaps persist. Among open-weight models, GLM-4.7 leads at 98.6\%, with several DeepSeek, Llama, Qwen and Mistral models also exceeding 93\%. The performance gap between open-weight and closed models is narrower than expected, with several lower-cost open-weight models exceeding 90\% accuracy. Petrophysics emerges as the most challenging domain across all models, while smaller models show wider performance variance. Residual length bias in the dataset (correct answers tend to be longer) is documented along with bias mitigation strategies applied during construction. The benchmark, evaluation code and results are publicly available.

翻译：本文提出了FormationEval，一个用于评估语言模型在石油地质科学与地下工程学科表现的开放式多项选择题基准。该数据集包含来自三个权威来源的505个问题，涵盖岩石物理学、石油地质学和油藏工程等七个领域，通过采用带有详细指令的推理模型和基于概念的方法构建，避免了逐字复制受版权保护的文本。每个问题均包含来源元数据，以支持可追溯性和审计。评估覆盖了来自OpenAI、Anthropic、Google、Meta等主要提供商及开源替代方案的72个模型。表现最佳的模型准确率超过97%，其中Gemini 3 Pro Preview达到99.8%，但层级和领域间的差距依然存在。在开源模型中，GLM-4.7以98.6%的准确率领先，多个DeepSeek、Llama、Qwen和Mistral模型也超过了93%。开源模型与闭源模型之间的性能差距比预期更小，多个低成本开源模型的准确率超过90%。岩石物理学是所有模型中最具挑战性的领域，而较小模型表现出更广泛的性能差异。本文记录了数据集中存在的残差长度偏差（正确答案往往更长），以及在构建过程中应用的偏差缓解策略。该基准、评估代码及结果均已公开提供。