Large language models (LLMs) have shown remarkable progress in coding and math problem-solving, but evaluation on advanced research-level problems in hard sciences remains scarce. To fill this gap, we present CMT-Benchmark, a dataset of 50 problems covering condensed matter theory (CMT) at the level of an expert researcher. Topics span analytical and computational approaches in quantum many-body, and classical statistical mechanics. The dataset was designed and verified by a panel of expert researchers from around the world. We built the dataset through a collaborative environment that challenges the panel to write and refine problems they would want a research assistant to solve, including Hartree-Fock, exact diagonalization, quantum/variational Monte Carlo, density matrix renormalization group (DMRG), quantum/classical statistical mechanics, and model building. We evaluate LLMs by programmatically checking solutions against expert-supplied ground truth. We developed machine-grading, including symbolic handling of non-commuting operators via normal ordering. They generalize across tasks too. Our evaluations show that frontier models struggle with all of the problems in the dataset, highlighting a gap in the physical reasoning skills of current LLMs. Notably, experts identified strategies for creating increasingly difficult problems by interacting with the LLMs and exploiting common failure modes. The best model, GPT5, solves 30\% of the problems; average across 17 models (GPT, Gemini, Claude, DeepSeek, Llama) is 11.4\pm2.1\%. Moreover, 18 problems are solved by none of the 17 models, and 26 by at most one. These unsolved problems span Quantum Monte Carlo, Variational Monte Carlo, and DMRG. Answers sometimes violate fundamental symmetries or have unphysical scaling dimensions. We believe this benchmark will guide development toward capable AI research assistants and tutors.

翻译：大型语言模型（LLM）在编程和数学问题求解方面已展现出显著进展，但在硬科学领域的高级研究级问题评估上仍十分匮乏。为填补这一空白，我们提出了CMT-Benchmark——一个包含50个凝聚态理论（CMT）问题的数据集，其难度达到专家研究者水平。主题涵盖量子多体与经典统计力学中的解析与计算方法。该数据集由来自全球的专家研究者小组设计并验证。我们通过协作环境构建数据集，要求专家小组撰写并完善他们期望研究助理能够解决的问题，包括哈特里-福克方法、精确对角化、量子/变分蒙特卡洛、密度矩阵重整化群（DMRG）、量子/经典统计力学以及模型构建。我们通过程序化比对专家提供的基准答案来评估LLM，开发了包含对非对易算子进行正规序符号化处理的机器评分系统，该系统也具备跨任务泛化能力。评估结果表明，前沿模型在应对数据集中所有问题时均表现挣扎，凸显了当前LLM在物理推理能力上的不足。值得注意的是，专家们通过交互式测试LLM并利用其常见失败模式，总结出构建渐进式难题的策略。最佳模型GPT5仅解决了30%的问题；17个模型（包括GPT、Gemini、Claude、DeepSeek、Llama）的平均解决率为11.4±2.1%。此外，有18个问题未被任何模型解决，26个问题最多仅被一个模型解决。这些未解问题涉及量子蒙特卡洛、变分蒙特卡洛及DMRG等领域。部分答案甚至违反基本对称性或呈现非物理的标度维度。我们相信该基准测试将为开发具备潜力的AI研究助手与教学工具提供指引。