The Smart Contract Weakness Classification Registry (SWC Registry) is a widely recognized list of smart contract weaknesses specific to the Ethereum platform. Despite the SWC Registry not being updated with new entries since 2020, the sustained development of smart contract analysis tools for detecting SWC-listed weaknesses highlights their ongoing significance in the field. However, evaluating these tools has proven challenging due to the absence of a large, unbiased, real-world dataset. To address this problem, we aim to build a large-scale SWC weakness dataset from real-world DApp projects. We recruited 22 participants and spent 44 person-months analyzing 1,199 open source audit reports from 29 security teams. In total, we identified 9,154 weaknesses and developed two distinct datasets, i.e., DAPPSCAN-SOURCE and DAPPSCAN-BYTECODE. The DAPPSCAN-SOURCE dataset comprises 39,904 Solidity files, featuring 1,618 SWC weaknesses sourced from 682 real-world DApp projects. However, the Solidity files in this dataset may not be directly compilable for further analysis. To facilitate automated analysis, we developed a tool capable of automatically identifying dependency relationships within DApp projects and completing missing public libraries. Using this tool, we created DAPPSCAN-BYTECODE dataset, which consists of 6,665 compiled smart contract with 888 SWC weaknesses. Based on DAPPSCAN-BYTECODE, we conducted an empirical study to evaluate the performance of state-of-the-art smart contract weakness detection tools. The evaluation results revealed sub-par performance for these tools in terms of both effectiveness and success detection rate, indicating that future development should prioritize real-world datasets over simplistic toy contracts.

翻译：智能合约漏洞分类注册表（SWC Registry）是广泛认可的针对以太坊平台的智能合约漏洞列表。尽管自2020年起SWC Registry未再更新新条目，但检测SWC所列漏洞的智能合约分析工具的持续发展，凸显了该列表在该领域的重要价值。然而，由于缺乏大规模、无偏见的真实世界数据集，评估这些工具一直颇具挑战。为解决这一问题，我们旨在从真实DApp项目中构建大规模SWC漏洞数据集。我们招募了22名参与者，投入44人月分析来自29个安全团队的1,199份开源审计报告。最终识别出9,154个漏洞，并构建了两个独立数据集：DAPPSCAN-SOURCE和DAPPSCAN-BYTECODE。DAPPSCAN-SOURCE数据集包含39,904个Solidity文件，涵盖来自682个真实DApp项目的1,618个SWC漏洞。但该数据集中的Solidity文件可能无法直接编译以进行后续分析。为促进自动化分析，我们开发了一种工具，可自动识别DApp项目中的依赖关系并补全缺失的公共库。利用该工具，我们创建了DAPPSCAN-BYTECODE数据集，包含6,665个已编译的智能合约及888个SWC漏洞。基于DAPPSCAN-BYTECODE，我们开展了实证研究，评估了当前最先进的智能合约漏洞检测工具的性能。评估结果显示，这些工具在有效性和成功检测率方面表现欠佳，表明未来开发应优先关注真实世界数据集，而非简化的玩具合约。