Selecting out-of-reach objects is a fundamental task in mixed reality (MR). Existing methods rely on a single cue or deterministically fuse multiple cues, leading to performance degradation when the dominant cue becomes unreliable. In this work, we introduce a probabilistic cue integration framework that enables flexible combination of multiple user-generated cues for intent inference. Inspired by natural grasping behavior, we instantiate the framework with pointing direction and grasp gestures as a new interaction technique, Point&Grasp. To this end, we collect the Out-of-Reach Grasping (ORG) dataset to train a robust likelihood model of the gestural cue, which captures grasping patterns not present in existing in-reach datasets. User studies demonstrate that our selection method with cue integration not only improves accuracy and speed over single-cue baselines, but also remains practically effective compared to state-of-the-art methods across various sources of ambiguity. The dataset and code are available at https://github.com/drlxj/point-and-grasp.
翻译:选择远距离物体是混合现实(MR)中的一项基础任务。现有方法依赖单一线索或确定性融合多个线索,当主导线索变得不可靠时会导致性能下降。本文引入一种概率线索融合框架,能够灵活结合多种用户生成线索进行意图推断。受自然抓握行为启发,我们以指向方向和抓握手势为具体实现,提出新型交互技术Point&Grasp。为此,我们采集Out-of-Reach Grasping (ORG)数据集,用于训练手势线索的稳健似然模型——该模型能够捕捉现有近距数据集未体现的抓取模式。用户研究表明,我们的线索融合选择方法不仅比单一线索基线方法提升了准确率和速度,而且在各种歧义场景中仍能保持与现有最优方法相当的实际效能。数据集与代码已开源至 https://github.com/drlxj/point-and-grasp。