Federated embodied agent learning protects the data privacy of individual visual environments by keeping data locally at each client (the individual environment) during training. However, since the local data is inaccessible to the server under federated learning, attackers may easily poison the training data of the local client to build a backdoor in the agent without notice. Deploying such an agent raises the risk of potential harm to humans, as the attackers may easily navigate and control the agent as they wish via the backdoor. Towards Byzantine-robust federated embodied agent learning, in this paper, we study the attack and defense for the task of vision-and-language navigation (VLN), where the agent is required to follow natural language instructions to navigate indoor environments. First, we introduce a simple but effective attack strategy, Navigation as Wish (NAW), in which the malicious client manipulates local trajectory data to implant a backdoor into the global model. Results on two VLN datasets (R2R and RxR) show that NAW can easily navigate the deployed VLN agent regardless of the language instruction, without affecting its performance on normal test sets. Then, we propose a new Prompt-Based Aggregation (PBA) to defend against the NAW attack in federated VLN, which provides the server with a ''prompt'' of the vision-and-language alignment variance between the benign and malicious clients so that they can be distinguished during training. We validate the effectiveness of the PBA method on protecting the global model from the NAW attack, which outperforms other state-of-the-art defense methods by a large margin in the defense metrics on R2R and RxR.

翻译：摘要：联邦式具身智能体学习通过将数据保留在各客户端（单个环境）本地进行训练，从而保护个体视觉环境的数据隐私。然而，由于联邦学习下服务器无法访问本地数据，攻击者可能轻易污染本地客户端的训练数据，在智能体中悄然植入后门。部署此类智能体会对人类构成潜在危害，因为攻击者可能通过后门随意导航和控制智能体。面向拜占庭鲁棒的联邦具身智能体学习，本文针对视觉-语言导航任务研究攻击与防御方法——该任务要求智能体遵循自然语言指令在室内环境中导航。首先，我们提出一种简单有效的攻击策略"如攻击者所愿的导航"（NAW），恶意客户端通过操纵局部轨迹数据为全局模型植入后门。在R2R和RxR两个VLN数据集上的实验表明，NAW能轻易操控已部署的VLN智能体无视语言指令导航，且不影响其在正常测试集上的性能。其次，我们提出新型提示基聚合（PBA）方法防御联邦VLN中的NAW攻击，该方法为服务器提供表征良性与恶意客户端间视觉-语言对齐差异的"提示"，从而在训练过程中区分二者。我们验证了PBA方法保护全局模型免受NAW攻击的有效性，其在R2R和RxR数据集上的防御指标显著优于其他最先进防御方法。