Learning task models of bimanual manipulation from human demonstration and their execution on a robot should take temporal constraints between actions into account. This includes constraints on (i) the symbolic level such as precedence relations or temporal overlap in the execution, and (ii) the subsymbolic level such as the duration of different actions, or their starting and end points in time. Such temporal constraints are crucial for temporal planning, reasoning, and the exact timing for the execution of bimanual actions on a bimanual robot. In our previous work, we addressed the learning of temporal task constraints on the symbolic level and demonstrated how a robot can leverage this knowledge to respond to failures during execution. In this work, we propose a novel model-driven approach for the combined learning of symbolic and subsymbolic temporal task constraints from multiple bimanual human demonstrations. Our main contributions are a subsymbolic foundation of a temporal task model that describes temporal nexuses of actions in the task based on distributions of temporal differences between semantic action keypoints, as well as a method based on fuzzy logic to derive symbolic temporal task constraints from this representation. This complements our previous work on learning comprehensive temporal task models by integrating symbolic and subsymbolic information based on a subsymbolic foundation, while still maintaining the symbolic expressiveness of our previous approach. We compare our proposed approach with our previous pure-symbolic approach and show that we can reproduce and even outperform it. Additionally, we show how the subsymbolic temporal task constraints can synchronize otherwise unimanual movement primitives for bimanual behavior on a humanoid robot.

翻译：从人类演示中学习双手操作的任务模型，并在机器人上执行时，需要考虑动作间的时间约束。这包括：（i）符号层面的约束，如执行顺序中的优先关系或时间重叠；（ii）子符号层面的约束，如不同动作的持续时间或其起始与终止时间点。这些时间约束对于时间规划、推理以及双手机器人上双手动作执行的精确时序至关重要。在先前工作中，我们解决了符号层面时间任务约束的学习问题，并展示了机器人如何利用该知识应对执行过程中的故障。本文提出一种新颖的模型驱动方法，用于从多个双手人类演示中联合学习符号与子符号时间任务约束。我们的主要贡献包括：基于语义动作关键点之间时间差异的分布，描述任务中动作时间纽结的子符号时间任务模型基础，以及基于模糊逻辑从该表示中推导符号时间任务约束的方法。这通过基于子符号基础整合符号与子符号信息，完善了我们先前关于学习全面时间任务模型的工作，同时保留了先前方法的符号表达能力。我们将所提方法与先前的纯符号方法进行比较，结果表明我们能够复现甚至超越其性能。此外，我们还展示了子符号时间任务约束如何将原本单手的运动基元同步为人形机器人的双手行为。