Diffusion-based action generation has become a foundational component of embodied AI, but its reliance on visual conditioning leaves deployed visuomotor policies vulnerable to adversarial manipulation. Most prior attacks focus on disruption: they perturb the observation stream to reduce task success or induce erratic behavior. We study a stronger threat, Test-time Adversarial Takeover (TAKO), in which an attacker obtains a real-time steering interface over a frozen robot policy and turns it into a remotely piloted instrument. TAKO learns a small vocabulary of reusable universal patches through differentiable diffusion inference; at test time, the attacker switches among these patches in the camera stream to compose attacker-chosen trajectories. This works because the perturbation acts on the visual conditioning pathway, where the induced bias can persist through iterative generative inference. We further show that the natural targeted baseline, target-policy matching, fails because the victim policy cannot reliably supervise itself on out-of-distribution target shifts. Across four tasks (2D manipulation, simulated aerial delivery, simulated ground navigation, and physical-world ground navigation), two visual encoders (ResNet-18 and EfficientNet-B0 + Transformer), and three generative inference families (DDPM, DDIM, and flow matching), human operators achieve 100\% takeover success on attacker-defined objectives in every evaluated setting. The project page is available at https://tako-attack.github.io.

翻译：基于扩散的动作生成已成为具身智能的基础组件，但其对视觉条件的依赖性使得部署的视觉运动策略易受对抗性操纵攻击。现有攻击主要聚焦于破坏性目标：通过扰动观测数据流降低任务成功率或诱发异常行为。本文研究一种更强的威胁——测试时对抗接管（TAKO），攻击者能够获取冻结机器人策略的实时转向接口，将其转变为远程操控工具。TAKO通过可微扩散推理学习小规模可复用通用补丁字典；在测试阶段，攻击者在相机数据流中切换这些补丁以合成攻击者指定的轨迹。该方法的有效性源于扰动作用于视觉条件路径，其诱导偏差可通过迭代生成推理持续传递。我们进一步证明，天然的目标导向基线——目标策略匹配——会因受害者策略无法可靠地在分布外目标偏移上自我监督而失效。在四项任务（二维操控、模拟空中递送、模拟地面导航及物理世界地面导航）、两种视觉编码器（ResNet-18与EfficientNet-B0+Transformer）以及三种生成推理框架（DDPM、DDIM与流匹配）中，人类操作员在所有评估场景下均实现了对攻击者定义目标的100%接管成功率。项目页面详见https://tako-attack.github.io。