Software obfuscation plays a crucial role in protecting intellectual property in software from reverse engineering attempts. While some obfuscation techniques originate from the obfuscation-reverse engineering arms race, others stem from different research areas, such as binary software exploitation. Return-oriented programming (ROP) gained popularity as one of the most effective exploitation techniques for memory error vulnerabilities. ROP interferes with our natural perception of a process control flow, inspiring us to repurpose ROP as a robust and effective form of software obfuscation. Although previous work already explores ROP's effectiveness as an obfuscation technique, evolving reverse engineering research raises the need for principled reasoning to understand the strengths and limitations of ROP-based mechanisms against man-at-the-end (MATE) attacks. To this end, we present ROPfuscator, a compiler-driven obfuscation pass based on ROP for any programming language supported by LLVM. We incorporate opaque predicates and constants and a novel instruction hiding technique to withstand sophisticated MATE attacks. More importantly, we introduce a realistic and unified threat model to thoroughly evaluate ROPfuscator and provide principled reasoning on ROP-based obfuscation techniques that answer to code coverage, incurred overhead, correctness, robustness, and practicality challenges.

翻译：软件混淆在保护软件知识产权免受逆向工程攻击中发挥着关键作用。虽然部分混淆技术源自混淆与逆向工程的对抗性演进，但另一些技术则源于不同研究领域，例如二进制软件利用技术。面向返回编程（ROP）凭借其应对内存错误漏洞的高效性，已成为最有效的利用技术之一。ROP干扰了人们对进程控制流的自然感知，这启发我们将ROP重新设计为一种鲁棒且高效的软件混淆形式。尽管已有研究探讨了ROP作为混淆技术的有效性，但逆向工程研究的持续演进要求我们通过原则性推理，深入理解基于ROP的机制在抵御末端攻击（MATE）时的优势与局限性。为此，我们提出ROPfuscator——一种基于LLVM、适用于其支持的任何编程语言的编译器驱动混淆模块。该模块整合了不透明谓词、常量以及一种新型指令隐藏技术，以抵御复杂的MATE攻击。更重要的是，我们引入了一个现实且统一的威胁模型，对ROPfuscator进行系统性评估，并从代码覆盖率、性能开销、正确性、鲁棒性及实用性等维度，为基于ROP的混淆技术提供了原则性推理依据。