We present a reconfigurable data glove design to capture different modes of human hand-object interactions, critical for training embodied AI agents for fine manipulation tasks. Sharing a unified backbone design that reconstructs hand gestures in real-time, our reconfigurable data glove operates in three modes for various downstream tasks with distinct features. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design is to minimize interference during complex hand movements. The Virtual Reality (VR) mode enables real-time interaction in a physically plausible fashion; a caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art Finite Element Method (FEM) simulator, the simulation mode collects a fine-grained 4D manipulation event: hand and object motions in 3D space and how the object's physical properties (e.g., stress, energy) change in accord with the manipulation in time. Of note, this glove system is the first to look into, through high-fidelity simulation, the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. Specifically, we evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses, respectively. Together, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both the physical and virtual environments, opening up new avenues to learning manipulation skills for embodied AI agents.

翻译：我们提出了一种可重构数据手套设计，用于捕捉人手与物体交互的不同模式，这对训练具身AI智能体完成精细操作任务至关重要。该手套系统采用统一的基础架构实现实时手部姿态重建，并支持三种针对不同下游任务具有独特功能的配置模式。在触觉感知模式下，手套系统通过由柔软且超薄压阻材料定制的力传感器聚合操作力；该设计旨在最大限度减少复杂手部运动过程中的干扰。虚拟现实模式下，系统能够以物理合理的方式实现实时交互；基于包围盒的算法通过检测碰撞事件来确定稳定抓取。通过利用最先进的有限元法仿真器，仿真模式可收集精细的四维操作事件：手部与物体在三维空间中的运动轨迹，以及物体物理属性（如应力、能量）随时间随操作变化的规律。值得注意的是，该手套系统是首个通过高保真仿真探究操作动作背后不可观测物理及因果因素的系统。在一系列实验中，我们从单个传感器和整体系统两个层面对数据手套进行表征。具体而言，我们通过以下方式评估系统的三种模式：（i）记录手部姿态及关联力数据，（ii）提升虚拟现实中的操作流畅性，（iii）生成多种工具使用的逼真仿真效果。总之，我们的可重构数据手套能够同时采集并重建物理环境与虚拟环境中精细的人手抓取数据，为具身AI智能体学习操作技能开辟了新途径。