Reinforcement learning (RL) has recently proven itself as a powerful instrument for solving complex problems and even surpassed human performance in several challenging applications. This signifies that RL algorithms can be used in the autonomous air combat problem, which has been studied for many years. The complexity of air combat arises from aggressive close-range maneuvers and agile enemy behaviors. In addition to these complexities, there may be uncertainties in real-life scenarios due to sensor errors, which prevent estimation of the actual position of the enemy. In this case, autonomous aircraft should be successful even in the noisy environments. In this study, we developed an air combat simulation, which provides noisy observations to the agents, therefore, make the air combat problem even more challenging. Thus, we present a state stacking method for noisy RL environments as a noise reduction technique. In our extensive set of experiments, the proposed method significantly outperforms the baseline algorithms in terms of the winning ratio, where the performance improvement is even more pronounced in the high noise levels. In addition, we incorporate a self-play scheme to our training process by periodically updating the enemy with a frozen copy of the training agent. By this way, the training agent performs air combat simulations to an enemy with smarter strategies, which improves the performance and robustness of the agents. In our simulations, we demonstrate that the self-play scheme provides important performance gains compared to the classical RL training.

翻译：强化学习（RL）近期已被证明是解决复杂问题的强大工具，甚至在多个具有挑战性的应用中超越了人类表现。这表明RL算法可用于自主空战问题，该问题已研究多年。空战的复杂性源于激烈的近距离机动和敏捷的敌方行为。除这些复杂性外，实际场景中可能因传感器误差存在不确定性，导致无法准确估计敌机的真实位置。在此情况下，自主飞行器即使在噪声环境中也应具备出色表现。本研究开发了一种空战仿真系统，为智能体提供含噪声的观测数据，从而进一步增加空战问题的难度。为此，我们针对噪声RL环境提出了一种状态堆叠方法作为降噪技术。在大量实验中，所提方法在胜率方面显著优于基线算法，且在高噪声水平下性能提升更为明显。此外，我们在训练过程中引入自对弈方案，定期用训练智能体的冻结副本更新敌方模型。通过这种方式，训练智能体与具备更智能策略的敌方进行空战仿真，从而提升智能体的性能和鲁棒性。仿真结果表明，与经典RL训练相比，自对弈方案带来了重要的性能提升。