Replicating a user's pose from only wearable sensors is important for many AR/VR applications. Most existing methods for motion tracking avoid environment interaction apart from foot-floor contact due to their complex dynamics and hard constraints. However, in daily life people regularly interact with their environment, e.g. by sitting on a couch or leaning on a desk. Using Reinforcement Learning, we show that headset and controller pose, if combined with physics simulation and environment observations can generate realistic full-body poses even in highly constrained environments. The physics simulation automatically enforces the various constraints necessary for realistic poses, instead of manually specifying them as in many kinematic approaches. These hard constraints allow us to achieve high-quality interaction motions without typical artifacts such as penetration or contact sliding. We discuss three features, the environment representation, the contact reward and scene randomization, crucial to the performance of the method. We demonstrate the generality of the approach through various examples, such as sitting on chairs, a couch and boxes, stepping over boxes, rocking a chair and turning an office chair. We believe these are some of the highest-quality results achieved for motion tracking from sparse sensor with scene interaction.

翻译：从可穿戴传感器复制用户姿态对许多增强现实/虚拟现实应用至关重要。由于交互动力学复杂且存在硬约束，大多数现有运动追踪方法（除脚-地接触外）避免与环境的交互。然而，日常生活中人们经常与环境交互，例如坐在沙发上或倚靠桌子。我们通过强化学习证明，结合物理仿真与环境观测的头显和控制器姿态，即使在高约束环境中也能生成逼真的全身姿态。物理仿真自动强制执行逼真姿态所需的各种约束，而非像许多运动学方法那样手动指定。这些硬约束使我们能够实现高质量的交互运动，避免穿透或接触滑动等典型伪影。我们讨论了三个对该方法性能至关重要的特征：环境表征、接触奖励和场景随机化。通过多种示例（如坐在椅子、沙发和箱子上，跨过箱子，摇晃椅子和转动办公椅）展示了该方法的通用性。我们认为这是目前从稀疏传感器进行场景交互运动追踪所达到的最高质量结果之一。