Existing approaches for autonomous control of pan-tilt-zoom (PTZ) cameras use multiple stages where object detection and localization are performed separately from the control of the PTZ mechanisms. These approaches require manual labels and suffer from performance bottlenecks due to error propagation across the multi-stage flow of information. The large size of object detection neural networks also makes prior solutions infeasible for real-time deployment in resource-constrained devices. We present an end-to-end deep reinforcement learning (RL) solution called Eagle to train a neural network policy that directly takes images as input to control the PTZ camera. Training reinforcement learning is cumbersome in the real world due to labeling effort, runtime environment stochasticity, and fragile experimental setups. We introduce a photo-realistic simulation framework for training and evaluation of PTZ camera control policies. Eagle achieves superior camera control performance by maintaining the object of interest close to the center of captured images at high resolution and has up to 17% more tracking duration than the state-of-the-art. Eagle policies are lightweight (90x fewer parameters than Yolo5s) and can run on embedded camera platforms such as Raspberry PI (33 FPS) and Jetson Nano (38 FPS), facilitating real-time PTZ tracking for resource-constrained environments. With domain randomization, Eagle policies trained in our simulator can be transferred directly to real-world scenarios.

翻译：摘要：现有自主控制云台变焦（PTZ）摄像机的方法采用多阶段流程，其中目标检测与定位独立于PTZ机制控制。这些方法需要人工标注，并因信息多阶段传递中的错误传播而面临性能瓶颈。目标检测神经网络的大规模参数使得先前方案无法在资源受限设备上实时部署。我们提出一种名为Eagle的端到端深度强化学习（RL）解决方案，旨在训练可直接以图像为输入控制PTZ摄像机的神经网络策略。由于标注成本、运行环境随机性及实验装置脆弱性，真实场景中的强化学习训练十分繁琐。我们引入一种照片级仿真框架，用于训练与评估PTZ摄像机控制策略。通过将感兴趣目标保持在高分辨率捕获图像的中心附近，Eagle实现了优越的摄像机控制性能，其跟踪持续时间较现有最优方法提升高达17%。Eagle策略轻量化（参数量较Yolo5s减少90倍），可在树莓派（33 FPS）和Jetson Nano（38 FPS）等嵌入式摄像机平台上运行，从而支持资源受限环境下的实时PTZ跟踪。结合领域随机化技术，在仿真器中训练的Eagle策略可直接迁移至真实场景。