Image-based motion prediction is one of the essential techniques for robot manipulation. Among the various prediction models, we focus on diffusion models because they have achieved state-of-the-art performance in various applications. In image-based motion prediction, diffusion models stochastically predict contextually appropriate motion by gradually denoising random Gaussian noise based on the image context. While diffusion models are able to predict various motions by changing the random noise, they sometimes fail to predict a contextually appropriate motion based on the image because the random noise is sampled independently of the image context. To solve this problem, we propose R2-Diff. In R2-Diff, a motion retrieved from a dataset based on image similarity is fed into a diffusion model instead of random noise. Then, the retrieved motion is refined through the denoising process of the diffusion model. Since the retrieved motion is almost appropriate to the context, it becomes easier to predict contextually appropriate motion. However, traditional diffusion models are not optimized to refine the retrieved motion. Therefore, we propose the method of tuning the hyperparameters based on the distance of the nearest neighbor motion among the dataset to optimize the diffusion model for refinement. Furthermore, we propose an image-based retrieval method to retrieve the nearest neighbor motion in inference. Our proposed retrieval efficiently computes the similarity based on the image features along the motion trajectory. We demonstrate that R2-Diff accurately predicts appropriate motions and achieves high task success rates compared to recent state-of-the-art models in robot manipulation.

翻译：图像驱动运动预测是机器人操作的核心技术之一。在众多预测模型中，我们聚焦于扩散模型，因其已在多种应用场景中取得最先进的性能。在图像驱动运动预测中，扩散模型通过根据图像上下文对随机高斯噪声逐步降噪，能够以随机方式预测符合上下文的运动。虽然扩散模型可通过改变随机噪声预测多种运动，但随机噪声的采样与图像上下文无关，导致模型有时无法预测出符合上下文的运动。为解决此问题，我们提出R2-Diff。该方法将基于图像相似性从数据集中检索到的运动替代随机噪声输入扩散模型，随后通过扩散模型的降噪过程对检索运动进行精细化处理。由于检索得到的运动已基本符合上下文，预测符合上下文运动的难度得以降低。然而传统扩散模型并非为细化检索运动而优化，为此我们提出基于数据集中最近邻运动距离对超参数进行调优的方法，使扩散模型适配细化任务。同时，我们提出一种推理阶段基于图像的检索方法，通过沿运动轨迹的图像特征高效计算相似度以获取最近邻运动。实验表明，与当前最先进的机器人操作模型相比，R2-Diff能准确预测合适运动，并实现更高的任务成功率。