We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy, privacy-preserving XR interactions. The source code and supplementary materials are available at: nanovis.org/ClickAIXR.html

翻译：我们提出ClickAIXR，一种新颖的设备端框架，用于实现与扩展现实（XR）中物体的多模态视觉-语言交互。与依赖云端AI（如ChatGPT）或基于注视的选择（如GazePointAR）的先前系统不同，ClickAIXR将设备端视觉-语言模型（VLM）与基于控制器的物体选择范式相结合，使用户能够在XR中精确点击真实物体。选定物体后，其图像由VLM在本地处理，以通过文本和语音两种方式回答自然语言问题。这种以物体为中心的交互减少了注视或语音单一界面固有的歧义性，并通过在设备端执行所有推理提升了透明度，从而解决了隐私和延迟问题。我们使用基于ONNX的本地VLM推理在Magic Leap SDK（C API）中实现了ClickAIXR。我们进行了一项用户研究，将ClickAIXR与Gemini 2.5 Flash和ChatGPT 5进行对比，评估了可用性、信任度和用户满意度。结果表明延迟适中，用户体验可接受。我们的发现展示了基于点击的物体选择与设备端AI相结合在推进可信、保护隐私的XR交互方面的潜力。源代码和补充材料可在以下网址获取：nanovis.org/ClickAIXR.html