In this paper, we investigate the operation of an aerial manipulator system, namely an Unmanned Aerial Vehicle (UAV) equipped with a controllable arm with two degrees of freedom to carry out actuation tasks on the fly. Our solution is based on employing a Q-learning method to control the trajectory of the tip of the arm, also called \textit{end-effector}. More specifically, we develop a motion planning model based on Time To Collision (TTC), which enables a quadrotor UAV to navigate around obstacles while ensuring the manipulator's reachability. Additionally, we utilize a model-based Q-learning model to independently track and control the desired trajectory of the manipulator's end-effector, given an arbitrary baseline trajectory for the UAV platform. Such a combination enables a variety of actuation tasks such as high-altitude welding, structural monitoring and repair, battery replacement, gutter cleaning, sky scrapper cleaning, and power line maintenance in hard-to-reach and risky environments while retaining compatibility with flight control firmware. Our RL-based control mechanism results in a robust control strategy that can handle uncertainties in the motion of the UAV, offering promising performance. Specifically, our method achieves 92\% accuracy in terms of average displacement error (i.e. the mean distance between the target and obtained trajectory points) using Q-learning with 15,000 episodes

翻译：本文研究了一种空中操纵系统的操作，即配备两自由度可控机械臂的无人机，用于在飞行中执行驱动任务。我们的解决方案基于采用Q学习方法控制机械臂末端（亦称末端执行器）的轨迹。具体而言，我们开发了一种基于碰撞时间（TTC）的运动规划模型，使四旋翼无人机能够在确保机械臂可达性的同时绕开障碍物。此外，我们利用基于模型的Q学习方法，在给定无人机平台任意基准轨迹的情况下，独立跟踪并控制机械臂末端执行器的期望轨迹。这种组合使得在难以到达及危险环境中执行高空焊接、结构监测与修复、电池更换、排水沟清洁、摩天大楼清洁及电力线路维护等多种驱动任务成为可能，同时保持与飞行控制固件的兼容性。基于强化学习的控制机制形成了一种鲁棒控制策略，能够处理无人机运动中的不确定性，展现出优越性能。具体而言，在15,000个训练回合的Q学习框架下，我们的方法在平均位移误差（即目标轨迹点与获取轨迹点之间的平均距离）上达到了92%的精度。