Recent advances in learning reusable motion priors have demonstrated their effectiveness in generating naturalistic behaviors. In this paper, we propose a new learning framework in this paradigm for controlling physics-based characters with significantly improved motion quality and diversity over existing state-of-the-art methods. The proposed method uses reinforcement learning (RL) to initially track and imitate life-like movements from unstructured motion clips using the discrete information bottleneck, as adopted in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure compresses the most relevant information from the motion clips into a compact yet informative latent space, i.e., a discrete space over vector quantized codes. By sampling codes in the space from a trained categorical prior distribution, high-quality life-like behaviors can be generated, similar to the usage of VQ-VAE in computer vision. Although this prior distribution can be trained with the supervision of the encoder's output, it follows the original motion clip distribution in the dataset and could lead to imbalanced behaviors in our setting. To address the issue, we further propose a technique named prior shifting to adjust the prior distribution using curiosity-driven RL. The outcome distribution is demonstrated to offer sufficient behavioral diversity and significantly facilitates upper-level policy learning for downstream tasks. We conduct comprehensive experiments using humanoid characters on two challenging downstream tasks, sword-shield striking and two-player boxing game. Our results demonstrate that the proposed framework is capable of controlling the character to perform considerably high-quality movements in terms of behavioral strategies, diversity, and realism. Videos, codes, and data are available at https://tencent-roboticsx.github.io/NCP/.

翻译：近年来，可复用运动先验学习方面的进步已证明其在生成自然行为方面的有效性。本文提出了一种基于该范式的新型学习框架，用于控制物理仿真角色，与现有最先进方法相比，显著提升了运动质量与多样性。该方法采用强化学习，通过离散信息瓶颈（如向量量化变分自编码器所采用的技术）初始跟踪并模仿非结构化运动片段中的逼真动作。该结构将运动片段中最相关的信息压缩至紧凑且富含信息的潜在空间（即基于向量量化编码的离散空间）。通过从训练好的类别先验分布中采样该空间内的编码，可生成高质量逼真行为，其原理类似于计算机视觉中VQ-VAE的应用。尽管该先验分布可通过编码器输出的监督进行训练，但其遵循数据集中的原始运动片段分布，可能导致本场景中的行为不平衡。为解决此问题，我们进一步提出名为先验偏移的技术，利用好奇心驱动的强化学习调整先验分布。实验证明，所得分布能提供充足的行为多样性，并显著促进下游任务中高层策略的学习。我们使用人体角色在两项具有挑战性的下游任务（剑盾格斗与双人拳击游戏）中进行了全面实验。结果表明，所提框架能够控制角色在行为策略、多样性与逼真度方面展现出极高品质的动作。相关视频、代码与数据集详见https://tencent-roboticsx.github.io/NCP/。