We present a novel approach for enhancing human-robot collaboration using physical interactions for real-time error correction of large language model (LLM) powered robots. Unlike other methods that rely on verbal or text commands, the robot leverages an LLM to proactively executes 6 DoF linear Dynamical System (DS) commands using a description of the scene in natural language. During motion, a human can provide physical corrections, used to re-estimate the desired intention, also parameterized by linear DS. This corrected DS can be converted to natural language and used as part of the prompt to improve future LLM interactions. We provide proof-of-concept result in a hybrid real+sim experiment, showcasing physical interaction as a new possibility for LLM powered human-robot interface.

翻译：我们提出了一种新颖方法，通过物理交互增强人机协作，实现对大型语言模型（LLM）驱动机器人的实时纠错。与依赖语音或文本指令的其他方法不同，该机器人利用LLM主动执行六自由度线性动态系统（DS）指令，这些指令基于自然语言场景描述生成。在运动过程中，人类可提供物理纠正，用于重新估计目标意图——该意图同样由线性DS参数化。修正后的DS可转换为自然语言，并作为提示词的一部分用于改进未来LLM交互。我们在混合现实+仿真实验中提供了概念验证结果，展示了物理交互作为LLM驱动人机接口的新可能性。