We present CineMPC, a complete cinematographic system that autonomously controls a drone to film multiple targets recording user-specified aesthetic objectives. Existing solutions in autonomous cinematography control only the camera extrinsics, namely its position, and orientation. In contrast, CineMPC is the first solution that includes the camera intrinsic parameters in the control loop, which are essential tools for controlling cinematographic effects like focus, depth-of-field, and zoom. The system estimates the relative poses between the targets and the camera from an RGB-D image and optimizes a trajectory for the extrinsic and intrinsic camera parameters to film the artistic and technical requirements specified by the user. The drone and the camera are controlled in a nonlinear Model Predicted Control (MPC) loop by re-optimizing the trajectory at each time step in response to current conditions in the scene. The perception system of CineMPC can track the targets' position and orientation despite the camera effects. Experiments in a photorealistic simulation and with a real platform demonstrate the capabilities of the system to achieve a full array of cinematographic effects that are not possible without the control of the intrinsics of the camera. Code for CineMPC is implemented following a modular architecture in ROS and released to the community.

翻译：我们提出CineMPC，一种完整的电影摄影系统，可自主控制无人机对多个目标进行拍摄，并满足用户指定的美学目标。现有自主电影摄影解决方案仅控制相机外部参数，即位置和朝向。相比之下，CineMPC是首个将相机内部参数纳入控制回路的方案，这些参数是控制对焦、景深、变焦等电影摄影效果的关键工具。该系统通过RGB-D图像估计目标与相机之间的相对位姿，并优化外部与内部相机参数的轨迹，以实现用户指定的艺术和技术要求。无人机与相机在非线性模型预测控制（MPC）回路中受控，该回路根据场景当前状态在每个时间步重新优化轨迹。CineMPC的感知系统能够跟踪目标的位置与朝向，不受相机特效影响。在逼真仿真平台和真实平台上的实验证明了该系统实现全面电影摄影效果的能力，而这些效果在缺乏内部参数控制的情况下无法达成。CineMPC的代码采用ROS模块化架构实现，并已开源发布。