Simulation enables scalable robot data collection, but raw 3D assets provide only geometry, lacking the semantic, interactive, and physical knowledge needed to specify where and how robots should act. In this work, we present AnnotateAnything, a general automatic annotation framework that converts passive 3D assets into manipulation-ready assets with structured, diverse, and executable manipulation labels. AnnotateAnything is built around two complementary pipelines. First, a unified visual-language annotation pipeline using vision-language reasoning to infer object semantics, interaction constraints, and 3D-grounded cues, providing human-prior guidance for identifying meaningful interaction regions. Second, a fully automatic and massively parallel physics annotation pipeline grounds these priors in each asset's geometry and physical constraints through candidate generation, geometry optimization and trajectory generation. This pipeline produces diverse and executable action annotations, including grasp poses, dexterous contacts, articulation waypoints, insertion directions, hanging affordances, and navigation targets. Using the generated annotations, we further build an asynchronous parallel simulation data-collection system across diverse objects, tasks, and robot embodiments. Experiments demonstrate that AnnotateAnything achieves superior annotation efficiency, data-collection efficiency, and task success rates over existing annotation and data-generation pipelines, while also supporting downstream tasks such as affordance detection, robotic VQA, and visual instruction finetuning. We provide project materials on the project page and plan to release the full code, annotations, and benchmark to facilitate future research. Videos, code, demo assets, and annotations are provided in supplementary materials Project page: https://tourmaline-caramel-169490.netlify.app.

翻译：仿真技术可支持大规模机器人数据采集，但原始3D资产仅包含几何信息，缺乏指定机器人操作位置与方式所需的语义、交互及物理知识。本文提出AnnotateAnything——一种通用自动标注框架，能将被动3D资产转化为具备结构化、多样化且可执行操作标签的操作就绪资产。该框架构建于两条互补流水线之上：首先，统一视觉-语言标注流水线通过视觉语言推理推断物体语义、交互约束及三维空间线索，为识别有意义的交互区域提供人类先验指导；其次，全自动大规模并行物理标注流水线通过候选生成、几何优化与轨迹生成，将上述先验锚定至各资产的几何结构与物理约束中，产生多样化且可执行的动作标注，包括抓取位姿、灵巧接触点、关节操作路径点、插入方向、悬挂功能区域及导航目标。基于生成的标注，我们进一步构建了面向多物体、多任务及多种机器人形态的异步并行仿真数据采集系统。实验表明，AnnotateAnything在标注效率、数据采集效率及任务成功率上均超越现有标注与数据生成流水线，同时支持功能区域检测、机器人视觉问答、视觉指令微调等下游任务。相关项目材料已发布于项目页面，完整代码、标注数据及基准测试集将开源以推动后续研究。视频、代码、演示资产及标注数据见补充材料，项目页面：https://tourmaline-caramel-169490.netlify.app