Malware authors often use cryptographic tools such as XOR encryption and block ciphers like AES to obfuscate part of the malware to evade detection. Use of cryptography may give the impression that these obfuscation techniques have some provable guarantees of success. In this paper, we take a closer look at the use of cryptographic tools to obfuscate malware. We first find that most techniques are easy to defeat (in principle), since the decryption algorithm and the key is shipped within the program. In order to clearly define an obfuscation technique's potential to evade detection we propose a principled definition of malware obfuscation, and then categorize instances of malware obfuscation that use cryptographic tools into those which evade detection and those which are detectable. We find that schemes that are hard to de-obfuscate necessarily rely on a construct based on environmental keying. We also show that cryptographic notions of obfuscation, e.g., indistinghuishability and virtual black box obfuscation, may not guarantee evasion detection under our model. However, they can be used in conjunction with environmental keying to produce hard to de-obfuscate versions of programs.

翻译：恶意软件作者常利用异或加密及AES等分组密码等密码学工具对部分恶意代码进行混淆，以规避检测。密码学的使用可能让人误以为这些混淆技术具备某种可证明的成功保障。本文深入探究了密码学工具在恶意软件混淆中的应用。我们首先发现，由于解密算法和密钥随程序一并打包，多数此类技术（原则上）易于破解。为清晰界定混淆技术规避检测的潜力，我们提出了恶意软件混淆的原则性定义，并将使用密码学工具的恶意软件混淆实例分为可规避检测与可被检测两类。研究显示，难以反混淆的方案必然依赖于基于环境密钥的构造。我们还证明，密码学意义上的混淆概念（如不可区分性与虚拟黑盒混淆）在我们模型下未必能保证规避检测。然而，这些概念可与环境密钥结合使用，生成难以反混淆的程序版本。