Research endeavors have been made in learning robust Multimodal Large Language Models (MLLMs) against jailbreak attacks. However, existing methods for improving MLLMs' robustness still face critical challenges: \ding{172} how to efficiently tune massive weight parameters and \ding{173} how to ensure robustness against attacks across both visual and textual modalities. To this end, we propose an \textbf{E}fficient \textbf{E}nd-to-end \textbf{A}dversarial \textbf{T}raining (E$^2$AT) framework for both visual and textual adversarial attacks. Specifically, for the visual aspect, E$^2$AT incorporates an efficient projector-based AT module that aligns the attack samples at the feature level. For training objectives, we propose a Dynamic Joint Multimodal Optimization (DJMO) strategy to enhance generalization ability against jailbreak attacks by dynamically adjusting weights between normal and adversarial objectives. Extensive experiments are conducted with five major jailbreak attack methods across three mainstream MLLMs. Results demonstrate that our E$^2$AT achieves the state-of-the-art performance, outperforming existing baselines by an average margin of 34\% across text and image modalities, while maintaining clean task performance. Furthermore, evaluations of real-world embodied intelligent systems highlight the practical applicability of E$^2$AT, paving the way for the development of more secure and reliable multimodal systems. Our code is available on \href{https://anonymous.4open.science/r/E2AT_568}{\textcolor{red}{https://anonymous.4open.science/r/E2AT\_568}}.

翻译：已有研究致力于学习能够抵御越狱攻击的鲁棒多模态大语言模型。然而，现有提升MLLM鲁棒性的方法仍面临关键挑战：① 如何高效微调海量权重参数；② 如何确保模型对视觉与文本双模态攻击的鲁棒性。为此，我们提出一种面向视觉与文本对抗攻击的**高效端到端对抗训练**框架。具体而言，在视觉方面，E$^2$AT引入了一个基于高效投影器的对抗训练模块，在特征层面对齐攻击样本。在训练目标上，我们提出**动态联合多模态优化**策略，通过动态调整正常目标与对抗目标间的权重，增强模型对越狱攻击的泛化能力。我们在三种主流MLLM上使用五种主要越狱攻击方法进行了大量实验。结果表明，我们的E$^2$AT取得了最先进的性能，在文本与图像模态上平均超越现有基线方法34%，同时保持了干净任务上的性能。此外，对现实世界具身智能系统的评估凸显了E$^2$AT的实际适用性，为开发更安全可靠的多模态系统铺平了道路。我们的代码公开于 \href{https://anonymous.4open.science/r/E2AT_568}{\textcolor{red}{https://anonymous.4open.science/r/E2AT\_568}}。