We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generation-and-evaluation pipeline. To validate the benchmark's reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexity-driven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs' literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.

翻译：我们提出SmartEval，这是一个系统性评估大语言模型（LLM）根据自然语言规范生成的Solidity智能合约质量的基准测试。SmartEval提供了包含9000份生成合约的语料库（这些合约与源自FSMSCG数据集且由专家编写的真实实现配对）、涵盖功能完整性、变量忠实度、状态机正确性、业务逻辑忠实度和代码质量五个维度的评估准则，以及可复现的生成与评估流水线。为验证基准测试的可靠性，我们开展了三项独立实证研究：一项五个条件的消融实验（每个条件N=300），用以隔离各流水线组件的贡献；一项由哥伦比亚大学三位博士研究员进行的人工专家评估，证实自动化评分与专家判断的偏差在0.34分以内；以及通过Slither静态分析器进行的外部安全分析，证实LLM审计器与基于规则的非LLM工具之间的一致性达79.4%。对9000份生成合约的系统性分析揭示了典型失效模式（逻辑缺失占35.3%，状态转换错误占23.4%，及复杂度驱动导致的性能退化），并量化了生成合约相比真实实现的+8.29综合评分优势，这归因于LLM遵循字面规范的行为。SmartEval为LLM智能合约合成质量的实证研究建立了可复现且经过验证的基础，所有数据、评估代码及生成合约均已公开发布。