Malware authors often use cryptographic tools such as XOR encryption and block ciphers like AES to obfuscate part of the malware to evade detection. Use of cryptography may give the impression that these obfuscation techniques have some provable guarantees of success. In this paper, we take a closer look at the use of cryptographic tools to obfuscate malware. We first find that most techniques are easy to defeat (in principle), since the decryption algorithm and the key is shipped within the program. In order to clearly define an obfuscation technique's potential to evade detection we propose a principled definition of malware obfuscation, and then categorize instances of malware obfuscation that use cryptographic tools into those which evade detection and those which are detectable. We find that schemes that are hard to de-obfuscate necessarily rely on a construct based on environmental keying. We also show that cryptographic notions of obfuscation, e.g., indistinghuishability and virtual black box obfuscation, may not guarantee evasion detection under our model. However, they can be used in conjunction with environmental keying to produce hard to de-obfuscate version of programs.

翻译：恶意软件作者常使用XOR加密和AES等分组密码等加密工具，对部分恶意软件进行混淆以逃避检测。密码学的使用可能让人产生这些混淆技术具有某种可证明成功保证的错觉。本文深入探讨了使用加密工具混淆恶意软件的实际情况。我们首先发现，由于解密算法和密钥均嵌入在程序内部，大多数技术（原则上）易于破解。为明确界定混淆技术逃避检测的能力，我们提出了恶意软件混淆的规范化定义，并将使用加密工具的恶意软件混淆实例分为可逃避检测与可被检测两类。研究发现，难以反混淆的方案必然依赖于基于环境密钥的构造。此外，我们还证明，密码学意义上的混淆概念（如不可区分性混淆和虚拟黑盒混淆）在我们的模型下可能无法保证逃避检测，但结合环境密钥使用时，可生成难以反混淆的程序版本。