Recent advances in foundation models, including large language models (LLMs), have created new opportunities to automate building energy modeling (BEM). However, systematic evaluation has remained challenging due to the absence of publicly available, task-specific datasets and standardized performance metrics. We present BEMEval, a benchmark framework designed to assess foundation models' performance across BEM tasks. The first benchmark in this suite, BEMEval-Doc2Schema, focuses on structured data extraction from building documentation, a foundational step toward automated BEM processes. BEMEval-Doc2Schema introduces the Key-Value Overlap Rate (KVOR), a metric that quantifies the alignment between LLM-generated structured outputs and ground-truth schema references. Using this framework, we evaluate two leading models (GPT-5 and Gemini 2.5) under zero-shot and few-shot prompting strategies across three datasets: HERS L100, NREL iUnit, and NIST NZERTF. Results show that Gemini 2.5 consistently outperforms GPT-5, and that few-shot prompts improve accuracy for both models. Performance also varies by schema: the EPC schema yields significantly higher KVOR scores than HPXML, reflecting its simpler and reduced hierarchical depth. By combining curated datasets, reproducible metrics, and cross-model comparisons, BEMEval-Doc2Schema establishes the first community-driven benchmark for evaluating LLMs in performing building energy modeling tasks, laying the groundwork for future research on AI-assisted BEM workflows.

翻译：基础模型（包括大语言模型）的最新进展为自动化建筑能耗建模（BEM）创造了新的机遇。然而，由于缺乏公开可用的任务特定数据集和标准化的性能指标，系统性评估一直面临挑战。我们提出了BEMEval，这是一个旨在评估基础模型在BEM任务中性能的基准框架。该套件中的首个基准，BEMEval-Doc2Schema，专注于从建筑文档中提取结构化数据，这是实现自动化BEM流程的基础步骤。BEMEval-Doc2Schema引入了键值重叠率（KVOR），这是一个量化LLM生成的结构化输出与真实模式参考之间对齐程度的指标。利用该框架，我们在三个数据集（HERS L100、NREL iUnit和NIST NZERTF）上，评估了两种领先模型（GPT-5和Gemini 2.5）在零样本和少样本提示策略下的表现。结果表明，Gemini 2.5始终优于GPT-5，并且少样本提示提高了两种模型的准确性。性能也因模式而异：EPC模式产生的KVOR分数显著高于HPXML，这反映了其更简单和更浅的层次深度。通过结合精选数据集、可复现的指标和跨模型比较，BEMEval-Doc2Schema建立了首个用于评估LLM执行建筑能耗建模任务的社区驱动基准，为未来AI辅助BEM工作流程的研究奠定了基础。