We introduce Multi-view Ancestral Sampling (MAS), a method for 3D motion generation, using 2D diffusion models that were trained on motions obtained from in-the-wild videos. As such, MAS opens opportunities to exciting and diverse fields of motion previously under-explored as 3D data is scarce and hard to collect. MAS works by simultaneously denoising multiple 2D motion sequences representing different views of the same 3D motion. It ensures consistency across all views at each diffusion step by combining the individual generations into a unified 3D sequence, and projecting it back to the original views. We demonstrate MAS on 2D pose data acquired from videos depicting professional basketball maneuvers, rhythmic gymnastic performances featuring a ball apparatus, and horse races. In each of these domains, 3D motion capture is arduous, and yet, MAS generates diverse and realistic 3D sequences. Unlike the Score Distillation approach, which optimizes each sample by repeatedly applying small fixes, our method uses a sampling process that was constructed for the diffusion framework. As we demonstrate, MAS avoids common issues such as out-of-domain sampling and mode-collapse. https://guytevet.github.io/mas-page/

翻译：本文提出多视角祖先采样 (MAS)，一种利用从野外视频中获取的运动数据训练的二维扩散模型生成三维运动的方法。由于三维数据稀缺且难以收集，MAS 为探索此前研究不足的众多新颖运动领域开辟了机遇。MAS 通过同时对表示同一三维运动的不同视角的多个二维运动序列进行去噪来实现。它在每个扩散步骤中通过将独立生成的序列合并为一个统一的三维序列，并将其投影回原始视角，从而确保所有视角之间的一致性。我们在从职业篮球战术动作、使用球类道具的韵律体操表演以及赛马等视频中获取的二维姿态数据上展示了 MAS 的效果。在每个领域中，三维运动捕捉均十分困难，而 MAS 仍能生成多样且逼真的三维序列。与分数蒸馏方法（通过反复施加微小修正优化每个样本）不同，我们的方法采用专为扩散框架构建的采样过程。实验证明，MAS 避免了常见问题，如域外采样和模式坍塌。https://guytevet.github.io/mas-page/