One promising use case of AI assistants is to help with complex procedures like cooking, home repair, and assembly tasks. Can we teach the assistant to interject after the user makes a mistake? This paper targets the problem of identifying ordering mistakes in assembly procedures. We propose a system that can detect ordering mistakes by utilizing a learned knowledge base. Our framework constructs a knowledge base with spatial and temporal beliefs based on observed mistakes. Spatial beliefs depict the topological relationship of the assembling components, while temporal beliefs aggregate prerequisite actions as ordering constraints. With an episodic memory design, our algorithm can dynamically update and construct the belief sets as more actions are observed, all in an online fashion. We demonstrate experimentally that our inferred spatial and temporal beliefs are capable of identifying incorrect orderings in real-world action sequences. To construct the spatial beliefs, we collect a new set of coarse-level action annotations for Assembly101 based on the positioning of the toy parts. Finally, we demonstrate the superior performance of our belief inference algorithm in detecting ordering mistakes on the Assembly101 dataset.
翻译:人工智能辅助程序最有前景的应用之一是在烹饪、家居维修和组装任务等复杂流程中提供帮助。我们能否教会辅助程序在用户犯错后进行干预?本文聚焦于识别组装过程中的顺序错误问题。我们提出一种利用学习构建的知识库来检测顺序错误的系统。该框架通过观测到的错误构建包含空间信念与时间信念的知识库:空间信念描述组装部件间的拓扑关系,时间信念则将前置动作聚合为顺序约束。基于情景记忆的设计,我们的算法能够以在线方式动态更新并构建信念集,随着更多动作被观测而持续完善。实验证明,我们推导出的空间与时间信念能有效识别真实动作序列中的错误顺序。为构建空间信念,我们基于玩具零件定位为Assembly101数据集采集了新的粗粒度动作标注。最终,我们展示了所提出的信念推理算法在Assembly101数据集上检测顺序错误时的卓越性能。