Augmented Reality (AR) assistance is increasingly used for supporting users with physical tasks like assembly and cooking. However, most systems rely on reactive responses triggered by user input, overlooking rich contextual and user-specific information. To address this, we present Satori, a novel AR system that proactively guides users by modeling both -- their mental states and environmental contexts. Satori integrates the Belief-Desire-Intention (BDI) framework with the state-of-the-art multi-modal large language model (LLM) to deliver contextually appropriate guidance. Our system is designed based on two formative studies involving twelve experts. We evaluated the system with a sixteen within-subject study and found that Satori matches the performance of designer-created Wizard-of-Oz (WoZ) systems, without manual configurations or heuristics, thereby improving generalizability, reusability, and expanding the potential of AR assistance.
翻译:增强现实(AR)辅助系统正日益广泛地应用于装配、烹饪等实体任务的用户支持。然而,现有系统大多依赖用户输入触发的被动响应,未能充分利用丰富的上下文信息与用户特定信息。为此,我们提出Satori——一种通过同步建模用户心理状态与环境上下文来实现主动引导的新型AR系统。Satori将信念-欲望-意图(BDI)框架与前沿多模态大语言模型(LLM)相结合,以提供符合情境的智能引导。本系统基于包含十二位专家参与的两项形成性研究设计而成。通过十六人次被试内实验评估表明:Satori在无需人工配置或启发式规则的情况下,其表现与设计者构建的 Wizard-of-Oz(WoZ)系统相当,从而提升了系统的泛化能力与可复用性,拓展了AR辅助技术的应用潜力。