In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks. To achieve various downstream tasks with distinct features, our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design minimizes interference during complex hand movements. The virtual reality (VR) mode enables real-time interaction in a physically plausible fashion: A caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art finite element method (FEM), the simulation mode collects data on fine-grained 4D manipulation events comprising hand and object motions in 3D space and how the object's physical properties (e.g., stress and energy) change in accordance with manipulation over time. Notably, the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. More specifically, we evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses, respectively. Based on these three modes, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments, thereby opening up new avenues for the learning of manipulation skills for embodied AI agents.

翻译：本研究提出一种可重构数据手套设计，用于捕捉人类手-物体交互的不同模式，这对训练具身人工智能（AI）代理完成精细操作任务至关重要。为实现具有不同特征的多样化下游任务，可重构数据手套以统一骨干设计为基础，实时重建手势，可在三种模式下运行。在触觉感知模式下，手套系统通过定制力传感器（采用柔软、薄型压阻材料制成）聚合操作力；该设计最大程度减少了复杂手部运动中的干扰。虚拟现实（VR）模式实现了物理上合理的实时交互：我们提出一种基于笼络的方法，通过检测碰撞事件来确定稳定抓取。利用先进的有界元方法（FEM），模拟模式可收集细粒度4D操作事件数据，包括手和物体在三维空间中的运动，以及物体物理属性（如应力和能量）随时间随操作的动态变化。值得注意的是，本文提出的手套系统是首个利用高保真模拟来探究操作动作背后不可观测的物理与因果因素的系统。通过一系列实验，我们从单个传感器和整体系统层面对数据手套进行了表征。具体而言，我们通过以下方式评估系统三种模式：(i) 记录手势及相关力值，(ii) 提高VR中的操作流畅性，(iii) 分别生成多种工具使用的逼真模拟效果。基于这三种模式，我们的可重构数据手套能够在物理与虚拟环境中收集并重建细粒度的人类抓取数据，从而为具身AI代理的操作技能学习开辟新途径。