State estimation is one of the greatest challenges for cloth manipulation due to cloth's high dimensionality and self-occlusion. Prior works propose to identify the full state of crumpled clothes by training a mesh reconstruction model in simulation. However, such models are prone to suffer from a sim-to-real gap due to differences between cloth simulation and the real world. In this work, we propose a self-supervised method to finetune a mesh reconstruction model in the real world. Since the full mesh of crumpled cloth is difficult to obtain in the real world, we design a special data collection scheme and an action-conditioned model-based cloth tracking method to generate pseudo-labels for self-supervised learning. By finetuning the pretrained mesh reconstruction model on this pseudo-labeled dataset, we show that we can improve the quality of the reconstructed mesh without requiring human annotations, and improve the performance of downstream manipulation task.

翻译：状态估计是布料操作中最具挑战性的问题之一，其原因在于布料的高维度和自遮挡特性。以往的研究通过仿真环境训练网格重建模型来识别褶皱布料的完整状态，但此类模型因布料仿真与现实世界的差异而易受仿真到现实差距的影响。本研究提出一种自监督方法，在现实世界中微调网格重建模型。由于现实世界中难以获取褶皱布料的完整网格，我们设计了一套特殊的数据采集方案和基于动作条件的模型驱动布料跟踪方法，为自监督学习生成伪标签。通过在伪标签数据集上微调预训练的网格重建模型，我们证明该方法无需人工标注即可提升重建网格的质量，并改善下游操作任务的性能。