Robust robot manipulation in unstructured environments often requires understanding object properties that extend beyond geometry, such as material or compliance-properties that can be challenging to infer using vision alone. Multimodal haptic sensing provides a promising avenue for inferring such properties, yet progress has been constrained by the lack of large, diverse, and realistic haptic datasets. In this work, we introduce the CLAMP device, a low-cost (<\$200) sensorized reacher-grabber designed to collect large-scale, in-the-wild multimodal haptic data from non-expert users in everyday settings. We deployed 16 CLAMP devices to 41 participants, resulting in the CLAMP dataset, the largest open-source multimodal haptic dataset to date, comprising 12.3 million datapoints across 5357 household objects. Using this dataset, we train a haptic encoder that can infer material and compliance object properties from multimodal haptic data. We leverage this encoder to create the CLAMP model, a visuo-haptic perception model for material recognition that generalizes to novel objects and three robot embodiments with minimal finetuning. We also demonstrate the effectiveness of our model in three real-world robot manipulation tasks: sorting recyclable and non-recyclable waste, retrieving objects from a cluttered bag, and distinguishing overripe from ripe bananas. Our results show that large-scale, in-the-wild haptic data collection can unlock new capabilities for generalizable robot manipulation. Website: https://emprise.cs.cornell.edu/clamp/

翻译：在非结构化环境中实现稳健的机器人操作通常需要理解超越几何形状的物体属性，例如材料或柔顺性——这些属性仅凭视觉往往难以推断。多模态触觉感知为推断此类属性提供了有前景的途径，然而其进展一直受限于缺乏大规模、多样化且真实的触觉数据集。本工作中，我们介绍了CLAMP设备，一种低成本（<200美元）的传感器化取物夹，专为从日常环境中的非专业用户处收集大规模真实世界多模态触觉数据而设计。我们向41名参与者分发了16台CLAMP设备，由此构建了CLAMP数据集——迄今为止最大的开源多模态触觉数据集，包含5357个家居物体的1230万个数据点。利用该数据集，我们训练了一个能够从多模态触觉数据中推断物体材料与柔顺性属性的触觉编码器。我们基于该编码器构建了CLAMP模型，一种用于材料识别的视觉-触觉感知模型，该模型能够泛化至新物体及三种机器人实体，仅需少量微调。我们还展示了该模型在三个真实世界机器人操作任务中的有效性：可回收与不可回收废物分类、从杂乱袋中抓取物体、以及区分过熟与成熟香蕉。我们的结果表明，大规模真实世界触觉数据采集能够为可泛化的机器人操作解锁新能力。项目网站：https://emprise.cs.cornell.edu/clamp/