Character animation in real-world scenarios necessitates a variety of constraints, such as trajectories, key-frames, interactions, etc. Existing methodologies typically treat single or a finite set of these constraint(s) as separate control tasks. They are often specialized, and the tasks they address are rarely extendable or customizable. We categorize these as solutions to the close-set motion control problem. In response to the complexity of practical motion control, we propose and attempt to solve the open-set motion control problem. This problem is characterized by an open and fully customizable set of motion control tasks. To address this, we introduce a new paradigm, programmable motion generation. In this paradigm, any given motion control task is broken down into a combination of atomic constraints. These constraints are then programmed into an error function that quantifies the degree to which a motion sequence adheres to them. We utilize a pre-trained motion generation model and optimize its latent code to minimize the error function of the generated motion. Consequently, the generated motion not only inherits the prior of the generative model but also satisfies the required constraints. Experiments show that we can generate high-quality motions when addressing a wide range of unseen tasks. These tasks encompass motion control by motion dynamics, geometric constraints, physical laws, interactions with scenes, objects or the character own body parts, etc. All of these are achieved in a unified approach, without the need for ad-hoc paired training data collection or specialized network designs. During the programming of novel tasks, we observed the emergence of new skills beyond those of the prior model. With the assistance of large language models, we also achieved automatic programming. We hope that this work will pave the way for the motion control of general AI agents.

翻译：现实场景中的角色动画需要满足多种约束条件，例如轨迹、关键帧、交互等。现有方法通常将单一或有限集合的此类约束视为独立的控制任务进行处理。这些方法往往具有专门性，所处理的任务难以扩展或定制。我们将此类方案归类为闭集运动控制问题的解决方案。针对实际运动控制的复杂性，我们提出并尝试解决开放集运动控制问题。该问题的特征在于其运动控制任务集合是开放且完全可定制的。为此，我们引入了一种新范式——可编程运动生成。在此范式中，任何给定的运动控制任务都被分解为原子约束的组合。这些约束随后被编程为一个误差函数，用于量化运动序列对约束的满足程度。我们利用预训练的运动生成模型，通过优化其潜在编码以最小化生成运动的误差函数。因此，生成的运动不仅继承了生成模型的先验特性，同时也满足了所需的约束条件。实验表明，在处理各类未见任务时，我们能够生成高质量的运动。这些任务涵盖通过运动动力学、几何约束、物理定律、与场景/物体或角色自身身体部位的交互等方式进行的运动控制。所有任务均通过统一方法实现，无需特设的配对训练数据收集或专门的网络设计。在对新任务进行编程的过程中，我们观察到先验模型之外的新技能涌现。借助大语言模型的协助，我们还实现了自动编程。我们希望这项工作能为通用智能体的运动控制开辟道路。