For natural human-robot interaction, a robot must understand human intent expressed not only through language but also through nonverbal signals such as gestures and gaze. However, current robot policies rely on language instructions as the sole interface for conveying intent, leaving nonverbal signals unused and placing the full burden of communication. In this work, we present EDITH, a robot framework that captures the human's nonverbal signals through continuous streams of first-person view and gaze from smart glasses, and uses them alongside language instructions as inputs to the robot policy. Our hardware system streams the human's first-person view, gaze, and speech to the robot in real time, transcribing the speech into language instructions. To handle these rich but noisy signals, we design a hierarchical policy in which a high-level policy infers the human's intent and produces a sequence of subtasks, where each subtask is represented as a fine-grained instruction paired with a keyframe that grounds the intent in the scene (e.g., the frame where the human points at the target object). A low-level policy then executes these subtasks. In our experiments on human-robot interactive tasks, EDITH enables the robot to act on the human's nonverbal signals even when intent is expressed only briefly, and significantly reduces user effort to convey intent compared to using language instructions alone. Visit our project page for source code and real-robot demo videos.

翻译：为了实现自然的人机交互，机器人不仅需要理解通过语言表达的人类意图，还需理解手势和注视等非语言信号。然而，当前的机器人策略仅将语言指令作为传达意图的唯一接口，忽视了非语言信号，并将全部沟通负担置于语言上。在本工作中，我们提出了EDITH机器人框架，该框架通过智能眼镜持续采集人类的第一人称视角和注视数据，并将这些数据与语言指令共同作为机器人策略的输入。我们的硬件系统实时传输人类的第一人称视角、注视及语音信息至机器人，并将语音转录为语言指令。为处理这些丰富但嘈杂的信号，我们设计了一种分层策略：高层策略推断人类意图并生成子任务序列，每个子任务表示为精细指令与关键帧的配对（例如，人类指向目标物体的帧），从而将意图锚定在场景中；低层策略则执行这些子任务。在人机交互实验任务中，EDITH使机器人能够基于人类短暂的非语言信号做出反应，并在传达意图时显著减少了用户的交互负担（相较于仅使用语言指令）。项目页面可获取源代码及真实机器人演示视频。