Recently, there has been a growing focus and interest in applying machine learning (ML) to the field of cybersecurity, particularly in malware detection and prevention. Several research works on malware analysis have been proposed, offering promising results for both academic and practical applications. In these works, the use of Generative Adversarial Networks (GANs) or Reinforcement Learning (RL) can aid malware creators in crafting metamorphic malware that evades antivirus software. In this study, we propose a mutation system to counteract ensemble learning-based detectors by combining GANs and an RL model, overcoming the limitations of the MalGAN model. Our proposed FeaGAN model is built based on MalGAN by incorporating an RL model called the Deep Q-network anti-malware Engines Attacking Framework (DQEAF). The RL model addresses three key challenges in performing adversarial attacks on Windows Portable Executable malware, including format preservation, executability preservation, and maliciousness preservation. In the FeaGAN model, ensemble learning is utilized to enhance the malware detector's evasion ability, with the generated adversarial patterns. The experimental results demonstrate that 100\% of the selected mutant samples preserve the format of executable files, while certain successes in both executability preservation and maliciousness preservation are achieved, reaching a stable success rate.

翻译：近年来，将机器学习（ML）应用于网络安全领域，特别是恶意软件检测与防御方面，日益受到关注和重视。已有多项关于恶意软件分析的研究工作被提出，在学术和实际应用中都取得了令人鼓舞的成果。在这些工作中，生成对抗网络（GAN）或强化学习（RL）的使用能够帮助恶意软件制作者制造出可逃避反病毒软件的变体恶意软件。本研究中，我们提出了一种结合GAN和RL模型的变异系统，以对抗基于集成学习的检测器，从而克服了MalGAN模型的局限性。我们提出的FeaGAN模型基于MalGAN构建，并整合了一个名为深度Q网络反恶意软件引擎攻击框架（DQEAF）的RL模型。该RL模型解决了对Windows可移植可执行文件进行对抗攻击时面临的三个关键挑战：格式保持、可执行性保持以及恶意性保持。在FeaGAN模型中，通过利用集成学习，使用生成的对抗模式来增强恶意软件检测器的逃逸能力。实验结果表明，100%选定的变异样本能够保持可执行文件的格式，同时在可执行性保持和恶意性保持方面也取得了一定的成功，达到了稳定的成功率。