Multimodal large language models (LMMs) excel in world knowledge and problem-solving abilities. Through the use of a world-facing camera and contextual AI, emerging smart accessories aim to provide a seamless interface between humans and LMMs. Yet, these wearable computing systems lack an understanding of the user's attention. We introduce GazeGPT as a new user interaction paradigm for contextual AI. GazeGPT uses eye tracking to help the LMM understand which object in the world-facing camera view a user is paying attention to. Using extensive user evaluations, we show that this gaze-contingent mechanism is a faster and more accurate pointing mechanism than alternatives; that it augments human capabilities by significantly improving their accuracy in a dog-breed classification task; and that it is consistently ranked as more natural than head- or body-driven selection mechanisms for contextual AI. Moreover, we prototype a variety of application scenarios that suggest GazeGPT could be of significant value to users as part of future AI-driven personal assistants.

翻译：多模态大语言模型（LMMs）在世界知识和问题解决能力方面表现出色。通过使用面向世界的摄像头和上下文AI，新兴的智能配件旨在为人机与LMMs之间提供无缝交互界面。然而，这些可穿戴计算系统缺乏对用户注意力的理解。我们提出GazeGPT作为一种新的上下文AI用户交互范式。GazeGPT利用眼动追踪帮助LMM理解用户正在关注面向世界摄像头视图中的哪个物体。通过大量用户评估，我们证明：这种基于注视的机制比替代方案更快速、更准确的指向机制；它通过显著提升用户在犬种分类任务中的准确性来增强人类能力；并且它始终被评为比基于头部或身体驱动的上下文AI选择机制更自然。此外，我们原型化了多种应用场景，表明GazeGPT作为未来AI驱动个人助手的一部分，可能对用户具有重要价值。