As robots become increasingly integrated into everyday tasks, their ability to perceive both the shape and properties of objects during in-hand manipulation becomes critical for adaptive and intelligent behavior. We present SemanticFeels, an extension of the NeuralFeels framework that integrates semantic labeling with neural implicit shape representation, from vision and touch. To illustrate its application, we focus on material classification: high-resolution Digit tactile readings are processed by a fine-tuned EfficientNet-B0 convolutional neural network (CNN) to generate local material predictions, which are then embedded into an augmented signed distance field (SDF) network that jointly predicts geometry and continuous material regions. Experimental results show that the system achieves a high correspondence between predicted and actual materials on both single- and multi-material objects, with an average matching accuracy of 79.87% across multiple manipulation trials on a multi-material object.

翻译：随着机器人日益融入日常任务，其在手内操作过程中感知物体形状与属性的能力对于实现自适应智能行为变得至关重要。本文提出SemanticFeels，该框架是NeuralFeels的扩展，它从视觉与触觉维度将语义标注与神经隐式形状表征相结合。为阐明其应用，我们聚焦于材料分类任务：高分辨率的Digit触觉读数通过微调的EfficientNet-B0卷积神经网络（CNN）进行处理，以生成局部材料预测，随后这些预测被嵌入到增强符号距离场（SDF）网络中，该网络联合预测几何形状与连续材料区域。实验结果表明，系统在单材料与多材料物体上均实现了预测材料与实际材料的高度对应，在对多材料物体进行的多次操作试验中平均匹配准确率达到79.87%。