User-generated cinematic creations are gaining popularity as our daily entertainment, yet it is a challenge to master cinematography for producing immersive contents. Many existing automatic methods focus on roughly controlling predefined shot types or movement patterns, which struggle to engage viewers with the circumstances of the actor. Real-world cinematographic rules show that directors can create immersion by comprehensively synchronizing the camera with the actor. Inspired by this strategy, we propose a deep camera control framework that enables actor-camera synchronization in three aspects, considering frame aesthetics, spatial action, and emotional status in the 3D virtual stage. Following rule-of-thirds, our framework first modifies the initial camera placement to position the actor aesthetically. This adjustment is facilitated by a self-supervised adjustor that analyzes frame composition via camera projection. We then design a GAN model that can adversarially synthesize fine-grained camera movement based on the physical action and psychological state of the actor, using an encoder-decoder generator to map kinematics and emotional variables into camera trajectories. Moreover, we incorporate a regularizer to align the generated stylistic variances with specific emotional categories and intensities. The experimental results show that our proposed method yields immersive cinematic videos of high quality, both quantitatively and qualitatively. Live examples can be found in the supplementary video.

翻译：用户生成的电影创作正日益成为我们日常娱乐的热门形式，然而掌握电影摄影技术以制作沉浸式内容仍具挑战。现有诸多自动化方法主要聚焦于粗略控制预定义的镜头类型或运动模式，难以使观众充分融入演员所处情境。现实世界的电影摄影规则表明，导演可通过将相机与演员全面同步来营造沉浸感。受此策略启发，我们提出一种深度相机控制框架，在三维虚拟场景中综合考虑画面美学、空间动作与情感状态，实现演员-相机三方面同步。遵循三分构图法则，本框架首先通过分析投影画面的自监督调节器，对初始相机位置进行美学化调整。随后设计生成对抗网络模型，基于演员的物理动作与心理状态对抗式合成细粒度相机运动，采用编码器-解码器生成器将运动学与情感变量映射为相机轨迹。此外，引入正则化器使生成的运动风格变化与特定情感类别及强度对齐。实验结果表明，所提方法在定量与定性评估中均能生成高质量沉浸式电影视频。动态示例可参见补充视频材料。