We present the Interactive Task Encoding System (ITES) for teaching robots to perform manipulative tasks. ITES is designed as an input system for the Learning-from-Observation (LfO) framework, which enables household robots to be programmed using few-shot human demonstrations without the need for coding. In contrast to previous LfO systems that rely solely on visual demonstrations, ITES leverages both verbal instructions and interaction to enhance recognition robustness, thus enabling multimodal LfO. ITES identifies tasks from verbal instructions and extracts parameters from visual demonstrations. Meanwhile, the recognition result was reviewed by the user for interactive correction. Evaluations conducted on a real robot demonstrate the successful teaching of multiple operations for several scenarios, suggesting the usefulness of ITES for multimodal LfO. The source code is available at https://github.com/microsoft/symbolic-robot-teaching-interface.

翻译：我们提出了交互式任务编码系统（ITES），用于教导机器人执行操作任务。ITES被设计为基于观察学习（LfO）框架的输入系统，使家用机器人能够通过少量人类示范进行编程，无需编码。与以往仅依赖视觉示范的LfO系统不同，ITES同时利用语言指令和交互来增强识别鲁棒性，从而实现多模态LfO。ITES从语言指令中识别任务，并从视觉示范中提取参数。同时，系统会将识别结果提交给用户进行审查，以便交互式修正。在真实机器人上进行的评估表明，ITES成功实现了多种场景下的多操作教学，验证了其用于多模态LfO的实用价值。源代码公开于https://github.com/microsoft/symbolic-robot-teaching-interface。