Machine learning is becoming increasingly popular as a go-to approach for many tasks due to its world-class results. As a result, antivirus developers are incorporating machine learning models into their products. While these models improve malware detection capabilities, they also carry the disadvantage of being susceptible to adversarial attacks. Although this vulnerability has been demonstrated for many models in white-box settings, a black-box attack is more applicable in practice for the domain of malware detection. We present a generator of adversarial malware examples using reinforcement learning algorithms. The reinforcement learning agents utilize a set of functionality-preserving modifications, thus creating valid adversarial examples. Using the proximal policy optimization (PPO) algorithm, we achieved an evasion rate of 53.84% against the gradient-boosted decision tree (GBDT) model. The PPO agent previously trained against the GBDT classifier scored an evasion rate of 11.41% against the neural network-based classifier MalConv and an average evasion rate of 2.31% against top antivirus programs. Furthermore, we discovered that random application of our functionality-preserving portable executable modifications successfully evades leading antivirus engines, with an average evasion rate of 11.65%. These findings indicate that machine learning-based models used in malware detection systems are vulnerable to adversarial attacks and that better safeguards need to be taken to protect these systems.

翻译：机器学习正因其世界级的表现而日益成为众多任务的首选方法。因此，防病毒开发者正将机器学习模型融入其产品中。尽管这些模型提升了恶意软件检测能力，但它们也面临着易受对抗攻击的弱点。虽然这一漏洞已在白盒环境下针对多种模型得到验证，但在恶意软件检测的实际应用中，黑盒攻击更具适用性。我们提出了一种利用强化学习算法生成对抗性恶意软件样本的生成器。该强化学习智能体采用一组保持功能的修改操作，从而生成有效的对抗样本。使用近端策略优化（PPO）算法，我们针对梯度提升决策树（GBDT）模型实现了53.84%的逃逸率。先前针对GBDT分类器训练的PPO智能体，在对抗基于神经网络的分类器MalConv时取得了11.41%的逃逸率，并针对顶级防病毒程序实现了平均2.31%的逃逸率。此外，我们发现随机应用保持功能的便携式可执行文件修改操作，能成功规避领先的防病毒引擎，平均逃逸率达11.65%。这些发现表明，恶意软件检测系统中基于机器学习的模型易受对抗攻击，且需要采取更完善的防护措施来保护这些系统。