Vision-driven autonomous flight and obstacle avoidance of Unmanned Aerial Vehicles (UAVs) along complex riverine environments for tasks like rescue and surveillance requires a robust control policy, which is yet difficult to obtain due to the shortage of trainable riverine environment simulators. To easily verify the vision-based navigation controller performance for the river following task before real-world deployment, we developed a trainable photo-realistic dynamics-free riverine simulation environment using Unity. In this paper, we address the shortcomings that vanilla Reinforcement Learning (RL) algorithm encounters in learning a navigation policy within this partially observable, non-Markovian environment. We propose a synergistic approach that integrates RL and Imitation Learning (IL). Initially, an IL expert is trained on manually collected demonstrations, which then guides the RL policy training process. Concurrently, experiences generated by the RL agent are utilized to re-train the IL expert, enhancing its ability to generalize to unseen data. By leveraging the strengths of both RL and IL, this framework achieves a faster convergence rate and higher performance compared to pure RL, pure IL, and RL combined with static IL algorithms. The results validate the efficacy of the proposed method in terms of both task completion and efficiency. The code and trainable environments are available.

翻译：视觉驱动的无人机（UAV）在复杂河流环境中执行救援、监控等任务时，需要具备鲁棒的控制策略以实现自主飞行与避障。然而，由于缺乏可训练的河流环境模拟器，获得此类策略极为困难。为了在真实部署前便捷验证基于视觉的导航控制器在沿河任务中的性能，我们利用Unity开发了一个可训练、照片级真实且无动力学的河流仿真环境。本文针对标准强化学习（RL）算法在部分可观测、非马尔可夫环境中学习导航策略的局限性，提出了一种融合强化学习与模仿学习（IL）的协同方法。首先，通过人工采集的演示数据训练IL专家模型，随后该模型指导RL策略的训练过程。同时，RL智能体生成的经历被用于重新训练IL专家，以增强其对未见数据的泛化能力。通过结合RL与IL的各自优势，该框架相比纯RL、纯IL以及RL与静态IL结合的算法，实现了更快的收敛速度与更高的性能。实验结果验证了该方法在任务完成度与效率两方面的有效性。相关代码与可训练环境已公开。