Federated embodied agent learning protects the data privacy of individual visual environments by keeping data locally at each client (the individual environment) during training. However, since the local data is inaccessible to the server under federated learning, attackers may easily poison the training data of the local client to build a backdoor in the agent without notice. Deploying such an agent raises the risk of potential harm to humans, as the attackers may easily navigate and control the agent as they wish via the backdoor. Towards Byzantine-robust federated embodied agent learning, in this paper, we study the attack and defense for the task of vision-and-language navigation (VLN), where the agent is required to follow natural language instructions to navigate indoor environments. First, we introduce a simple but effective attack strategy, Navigation as Wish (NAW), in which the malicious client manipulates local trajectory data to implant a backdoor into the global model. Results on two VLN datasets (R2R and RxR) show that NAW can easily navigate the deployed VLN agent regardless of the language instruction, without affecting its performance on normal test sets. Then, we propose a new Prompt-Based Aggregation (PBA) to defend against the NAW attack in federated VLN, which provides the server with a ''prompt'' of the vision-and-language alignment variance between the benign and malicious clients so that they can be distinguished during training. We validate the effectiveness of the PBA method on protecting the global model from the NAW attack, which outperforms other state-of-the-art defense methods by a large margin in the defense metrics on R2R and RxR.

翻译：摘要：联邦式具身智能体学习通过将各客户端（独立环境）的数据本地化存储于训练过程中，从而保护个体视觉环境的数据隐私。然而，由于联邦学习机制下服务器无法访问本地数据，攻击者可轻易污染本地客户端训练数据，在智能体中无声植入后门。部署此类智能体将对人类构成潜在危害风险，因攻击者可经由后门随意导航与控制该智能体。本文针对拜占庭鲁棒的联邦具身智能体学习展开研究，聚焦视觉-语言导航（VLN）任务的攻防问题——该任务要求智能体遵循自然语言指令在室内环境中导航。首先，我们提出一种简单有效的攻击策略"随心导航"（NAW）：恶意客户端通过操纵本地轨迹数据向全局模型植入后门。在R2R与RxR两个VLN数据集上的实验表明，NAW可轻松操控已部署的VLN智能体（不论语言指令为何），且不影响其在正常测试集上的表现。随后，我们提出基于提示的聚合方法（PBA）以防御联邦VLN中的NAW攻击：该方法通过向服务器提供"提示"来表征良性客户端与恶意客户端之间的视觉-语言对齐差异，从而在训练过程中实现二者区分。我们验证了PBA方法在保护全局模型免受NAW攻击方面的有效性，在R2R与RxR数据集上的防御指标中，该方法以显著优势超越其他先进防御方法。