Physical Human-Scene Interaction (HSI) plays a crucial role in numerous applications. However, existing HSI techniques are limited to specific object dynamics and privileged information, which prevents the development of more comprehensive applications. To address this limitation, we introduce HumanVLA for general object rearrangement directed by practical vision and language. A teacher-student framework is utilized to develop HumanVLA. A state-based teacher policy is trained first using goal-conditioned reinforcement learning and adversarial motion prior. Then, it is distilled into a vision-language-action model via behavior cloning. We propose several key insights to facilitate the large-scale learning process. To support general object rearrangement by physical humanoid, we introduce a novel Human-in-the-Room dataset encompassing various rearrangement tasks. Through extensive experiments and analysis, we demonstrate the effectiveness of the proposed approach.

翻译：物理人-场景交互（HSI）在众多应用中发挥着关键作用。然而，现有HSI技术受限于特定的物体动力学和特权信息，阻碍了更全面应用的发展。为解决这一局限，我们提出了HumanVLA，用于实现由实际视觉和语言引导的通用物体重排。我们采用师生框架来构建HumanVLA。首先，使用目标条件强化学习和对抗运动先验训练一个基于状态的教师策略。随后，通过行为克隆将其蒸馏到一个视觉-语言-动作模型中。我们提出了若干关键见解以促进大规模学习过程。为支持物理人形机器人进行通用物体重排，我们引入了一个新颖的“人在房间中”数据集，该数据集涵盖了多种重排任务。通过大量实验与分析，我们验证了所提方法的有效性。