In order to build artificial intelligence systems that can perceive and reason with human behavior in the real world, we must first design models that conduct complex spatio-temporal reasoning over motion sequences. Moving towards this goal, we propose the HumanMotionQA task to evaluate complex, multi-step reasoning abilities of models on long-form human motion sequences. We generate a dataset of question-answer pairs that require detecting motor cues in small portions of motion sequences, reasoning temporally about when events occur, and querying specific motion attributes. In addition, we propose NSPose, a neuro-symbolic method for this task that uses symbolic reasoning and a modular design to ground motion through learning motion concepts, attribute neural operators, and temporal relations. We demonstrate the suitability of NSPose for the HumanMotionQA task, outperforming all baseline methods.

翻译：为了构建能够感知并在现实世界中推理人类行为的人工智能系统，我们必须首先设计能够对运动序列进行复杂时空推理的模型。为实现这一目标，我们提出了HumanMotionQA任务，用于评估模型对长时间人体运动序列执行复杂、多步推理的能力。我们生成了一个问答对数据集，该数据集要求检测运动序列中微小部分的运动线索，对事件发生的时间进行时序推理，并查询特定的运动属性。此外，我们提出了NSPose——一种面向该任务的神经符号方法，该方法通过符号推理和模块化设计，借助学习运动概念、属性神经算子以及时序关系来锚定运动。我们证明了NSPose在HumanMotionQA任务上的适用性，其表现优于所有基线方法。