Simulation-based data generation has become a dominant paradigm for training robotic manipulation policies, yet existing platforms do not incorporate object affordance information into trajectory generation. As a result, tasks requiring precise interaction with specific functional regions--grasping a mug by its handle, pouring from a cup's rim, or hanging a mug on a hook--cannot be automatically generated with semantically correct trajectories. We introduce AffordSim, the first simulation framework that integrates open-vocabulary 3D affordance prediction into the manipulation data generation pipeline. AffordSim uses our VoxAfford model, an open-vocabulary 3D affordance detector that enhances MLLM output tokens with multi-scale geometric features, to predict affordance maps on object point clouds, guiding grasp pose estimation toward task-relevant functional regions. Built on NVIDIA Isaac Sim with cross-embodiment support (Franka FR3, Panda, UR5e, Kinova), VLM-powered task generation, and novel domain randomization using DA3-based 3D Gaussian reconstruction from real photographs, AffordSim enables automated, scalable generation of affordance-aware manipulation data. We establish a benchmark of 50 tasks across 7 categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) and evaluate 4 imitation learning baselines (BC, Diffusion Policy, ACT, Pi 0.5). Our results reveal that while grasping is largely solved (53-93% success), affordance-demanding tasks such as pouring into narrow containers (1-43%) and mug hanging (0-47%) remain significantly more challenging for current imitation learning methods, highlighting the need for affordance-aware data generation. Zero-shot sim-to-real experiments on a real Franka FR3 validate the transferability of the generated data.

翻译：基于仿真的数据生成已成为训练机器人操控策略的主流范式，然而现有平台未将物体可操作信息融入轨迹生成。因此，需要与特定功能区域精确交互的任务——例如抓取杯柄、从杯沿倒水或将杯子挂在挂钩上——无法自动生成语义正确的轨迹。我们提出AffordSim，这是首个将开放词汇3D可操作预测集成到操控数据生成流水线中的仿真框架。AffordSim利用我们提出的VoxAfford模型（一种增强多尺度几何特征的多模态大语言模型输出令牌的开放词汇3D可操作检测器），在物体点云上预测可操作图，从而引导抓取姿态估计转向任务相关的功能区域。该框架基于NVIDIA Isaac Sim构建，支持多形态控制（Franka FR3、Panda、UR5e、Kinova），结合视觉语言模型驱动的任务生成，以及利用基于DA3的真实照片3D高斯重建的新型域随机化技术，实现了可操作感知操控数据的自动化、可扩展生成。我们建立了涵盖7个类别（抓取、放置、堆叠、推拉、倾倒、挂杯、长时序组合）共50个任务的基准测试，并评估了4种模仿学习基线方法（行为克隆、扩散策略、ACT、Pi 0.5）。实验结果表明：尽管抓取任务已基本解决（成功率53-93%），但需精细可操作感知的任务（如向窄口容器倾倒，成功率1-43%；挂杯，成功率0-47%）对当前模仿学习方法仍构成显著挑战，这凸显了可操作感知数据生成的必要性。在真实Franka FR3上的零样本仿真到现实迁移实验验证了所生成数据的可迁移性。