Background: Existing MRI LLM benchmarks rely mainly on review-book multiple-choice questions, where top proprietary models already score highly, limiting discrimination. No systematic benchmark has evaluated vendor-specific scanner operational knowledge central to research MRI practice. Purpose: We developed MRI-Eval, a tiered benchmark for relative model comparison on MRI physics and GE scanner operations knowledge using primary multiple-choice questions (MCQ), with stem-only and primed diagnostic conditions as complementary analyses. Methods: MRI-Eval includes 1365 scored items across nine categories and three difficulty tiers from textbooks, GE scanner manuals, programming course materials, and expert-generated questions. Five model families were evaluated (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, Llama 3.3 70B). MCQ was primary; stem-only removed options and used an independent LLM judge; primed stem-only tested responses to incorrect user claims. Results: Overall MCQ accuracy was 93.2% to 97.1%. GE scanner operations was the lowest category for every model (88.2% to 94.6%). In stem-only, frontier-model accuracy fell to 58.4% to 61.1%, and Llama 3.3 70B fell to 37.1%; GE scanner operations stem-only accuracy was 13.8% to 29.8%. Conclusion: High MCQ performance can mask weak free-text recall, especially for vendor-specific operational knowledge. MRI-Eval is most informative as a relative comparison benchmark rather than an absolute competency measure and supports caution in using raw LLM outputs for GE-specific protocol guidance.

翻译：背景：现有MRI领域的LLM基准主要依赖基于复习教材的多项选择题，顶尖商业模型在该类测试中已取得高分，导致区分度有限。尚无系统性基准评估过对研究型MRI实践中至关重要的供应商专用扫描仪操作知识。目的：我们开发了MRI-Eval，这是一个分层基准，通过基于主干题干与预置诊断条件的互补分析，使用原创多项选择题对MRI物理与GE扫描仪操作知识进行相对模型比较。方法：MRI-Eval包含1365道评分题，涵盖九大类别与三个难度层级，题目来源包括教科书、GE扫描仪手册、编程课程材料及专家生成题目。评估了五个模型家族（GPT-5.4、Claude Opus 4.6、Claude Sonnet 4.6、Gemini 2.5 Pro、Llama 3.3 70B）。以多项选择题为主，题干剥离模式移除选项并由独立LLM法官评判，预置题干剥离模式则测试对用户错误主张的响应。结果：总体多项选择题准确率为93.2%至97.1%。所有模型中最低的分类均为GE扫描仪操作（88.2%至94.6%）。题干剥离模式下，前沿模型准确率降至58.4%至61.1%，Llama 3.3 70B降至37.1%；GE扫描仪操作在题干剥离模式下的准确率仅为13.8%至29.8%。结论：高多项选择题表现可能掩盖弱自由文本回忆能力，尤其对供应商专用操作知识而言。MRI-Eval作为相对比较基准而非绝对能力衡量工具最具信息量，并提示对直接使用LLM输出进行GE专用协议指引需保持审慎。