Binary analysis is an important capability required for many security and software engineering applications. Consequently, there are many binary analysis techniques and tools with varied capabilities. However, testing these tools requires a large, varied binary dataset with corresponding source-level information. In this paper, we present Cornucopia, an architecture agnostic automated framework that can generate a plethora of binaries from corresponding program source by exploiting compiler optimizations and feedback-guided learning. Our evaluation shows that Cornucopia was able to generate 309K binaries across four architectures (x86, x64, ARM, MIPS) with an average of 403 binaries for each program and outperforms Bintuner, a similar technique. Our experiments revealed issues with the LLVM optimization scheduler resulting in compiler crashes ($\sim$300). Our evaluation of four popular binary analysis tools Angr, Ghidra, Idapro, and Radare, using Cornucopia generated binaries, revealed various issues with these tools. Specifically, we found 263 crashes in Angr and one memory corruption issue in Idapro. Our differential testing on the analysis results revealed various semantic bugs in these tools. We also tested machine learning tools, Asmvec, Safe, and Debin, that claim to capture binary semantics and show that they perform poorly (For instance, Debin F1 score dropped to 12.9% from reported 63.1%) on Cornucopia generated binaries. In summary, our exhaustive evaluation shows that Cornucopia is an effective mechanism to generate binaries for testing binary analysis techniques effectively.

翻译：二进制分析是许多安全和软件工程应用需要的重要能力。 因此, 许多二进制分析技术和工具都具有不同的能力。 然而, 测试这些工具需要大量、 多种多样的二进制数据集, 以及相应的源级信息 。 在本文中, 我们展示了Cornucopia, 这个建筑的不可知性自动框架, 通过利用编译器优化和反馈指导学习, 可以从相应的程序源产生大量的二进制书。 我们的评估显示, Cornucopia 能够在四个结构( X86, x64, ARM, MIPS) 中生成 309K 二进制二进制二进制二进制书, 每个程序平均403个二进制二进制二进制二进制二进制二, 类似技术。 我们的实验揭示了LLVM 优化调度仪的问题, 导致编译者崩溃( $300) 。 我们对四个流行的二进制分析工具 Angr, Ghidra, Idapro, 和Radee, 揭示了这些工具的系统测试结果。