Malicious transactions on smart contracts cause billions of dollars in losses. To evaluate their detectors, most datasets copy the attack transactions that public incident databases cite and pair them with random transactions. However, these databases never check their citations against execution, and random transactions rarely resemble attacks. We present TXBENCH, a benchmark that verifies both attacks and legitimate transactions against their execution. TXBENCH labels a cited transaction as an attack only when its execution takes value from others and independent evidence ties it to its incident. Its legitimate transactions perform the same operations as attacks and carry positive evidence of legitimacy. Applying these checks to two public databases, we find that they share only 10.6% of their 1,850 cited transactions and that only 1,012 are verifiable attacks, worth $868.2 million. A blind manual review confirms 98/100 sampled attacks and 100/100 legitimate transactions.
翻译:暂无翻译