Clothes manipulation, such as folding or hanging, is a critical capability for home service robots. Despite recent advances, most existing methods remain limited to specific clothes types and tasks, due to the complex, high-dimensional geometry of clothes. This paper presents CLothes mAnipulation with Semantic keyPoints (CLASP), which aims at general-purpose clothes manipulation over diverse clothes types, T-shirts, shorts, skirts, long dresses, ..., as well as different tasks, folding, flattening, hanging, .... The core idea of CLASP is semantic keypoints-e.g., ''left sleeve'' and ''right shoulder''-a sparse spatial-semantic representation, salient for both perception and action. Semantic keypoints of clothes can be reliably extracted from RGB-D images and provide an effective representation for a wide range of clothes manipulation policies. CLASP uses semantic keypoints as an intermediate representation to connect high-level task planning and low-level action execution. At the high level, it exploits vision language models (VLMs) to predict task plans over the semantic keypoints. At the low level, it executes the plans with the help of a set of pre-built manipulation skills conditioned on the keypoints. Extensive simulation experiments show that CLASP outperforms state-of-the-art baseline methods on multiple tasks across diverse clothes types, demonstrating strong performance and generalization. Further experiments with a Franka dual-arm system on four distinct tasks-folding, flattening, hanging, and placing-confirm CLASP's performance on real-life clothes manipulation.

翻译：衣物操控（如折叠或悬挂）是家庭服务机器人的关键能力。尽管近期取得进展，但由于衣物复杂的高维几何特性，现有方法大多仍局限于特定衣物类型和任务。本文提出基于语义关键点的衣物操控方法（CLASP），旨在实现对多种衣物类型（T恤、短裤、裙装、长礼服等）及不同任务（折叠、铺平、悬挂等）的通用操控。CLASP的核心思想是语义关键点（如"左袖口"和"右肩部"）——一种对感知与动作均具有显著意义的稀疏空间语义表征。衣物语义关键点可从RGB-D图像中可靠提取，并为各类衣物操控策略提供有效表征。CLASP将语义关键点作为中间表征连接高层任务规划与底层动作执行：在高层，利用视觉语言模型（VLMs）预测基于语义关键点的任务规划；在底层，借助一组基于关键点预构建的操控技能执行规划。大量仿真实验表明，CLASP在多种衣物类型的多项任务上优于现有基线方法，展现出卓越性能与泛化能力。通过Franka双臂系统在折叠、铺平、悬挂和放置四项任务中的进一步实验，验证了CLASP在真实衣物操控场景中的有效性。