Diffusion policies are becoming mainstream in robotic manipulation but suffer from hard negative class imbalance due to uniform sampling and lack of sample difficulty awareness, leading to slow training convergence and frequent inference timeout failures. We propose VADF (Vision-Adaptive Diffusion Policy Framework), a vision-driven dual-adaptive framework that significantly reduces convergence steps and achieves early success in inference, with model-agnostic design enabling seamless integration into any diffusion policy architecture. During training, we introduce Adaptive Loss Network (ALN), a lightweight MLP-based loss predictor that quantifies per-step sample difficulty in real time. Guided by hard negative mining, it performs weighted sampling to prioritize high-loss regions, enabling adaptive weight updates and faster convergence. In inference, we design the Hierarchical Vision Task Segmenter (HVTS), which decomposes high-level task instructions into multi-stage low-level sub-instructions based on visual input. It adaptively segments action sequences into simple and complex subtasks by assigning shorter noise schedules with longer direct execution sequences to simple actions, and longer noise steps with shorter execution sequences to complex ones, thereby dramatically reducing computational overhead and significantly improving the early success rate.

翻译：扩散策略在机器人操作领域逐渐成为主流，但由于均匀采样且缺乏样本难度感知，导致训练收敛缓慢及推理超时故障频发。为此，我们提出VADF（视觉自适应扩散策略框架），一种视觉驱动的双自适应框架，显著减少收敛步数并在推理阶段实现早期成功。该框架具有模型无关特性，可无缝集成至任意扩散策略架构。在训练阶段，我们引入自适应损失网络（ALN），一种基于MLP的轻量级损失预测器，可实时量化每步样本难度。基于难负样本挖掘策略，该网络对高损失区域进行加权采样以优先处理，实现自适应权重更新并加速收敛。在推理阶段，我们设计了分层视觉任务分割器（HVTS），通过视觉输入将高层任务指令分解为多阶段低层子指令，自适应地将动作序列划分为简单与复杂子任务：为简单动作分配更短的噪声调度和更长的直接执行序列，为复杂动作分配更长的噪声步数和更短的执行序列，从而显著降低计算开销，大幅提升早期成功率。