Over the past few years, the emergence of backdoor attacks has presented significant challenges to deep learning systems, allowing attackers to insert backdoors into neural networks. When data with a trigger is processed by a backdoor model, it can lead to mispredictions targeted by attackers, whereas normal data yields regular results. The scope of backdoor attacks is expanding beyond computer vision and encroaching into areas such as natural language processing and speech recognition. Nevertheless, existing backdoor defense methods are typically tailored to specific data modalities, restricting their application in multimodal contexts. While multimodal learning proves highly applicable in facial recognition, sentiment analysis, action recognition, visual question answering, the security of these models remains a crucial concern. Specifically, there are no existing backdoor benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal backdoor, we introduce BackdoorMBTI, the first backdoor learning toolkit and benchmark designed for multimodal evaluation across three representative modalities from eleven commonly used datasets. BackdoorMBTI provides a systematic backdoor learning pipeline, encompassing data processing, data poisoning, backdoor training, and evaluation. The generated poison datasets and backdoor models enable detailed evaluation of backdoor defenses. Given the diversity of modalities, BackdoorMBTI facilitates systematic evaluation across different data types. Furthermore, BackdoorMBTI offers a standardized approach to handling practical factors in backdoor learning, such as issues related to data quality and erroneous labels. We anticipate that BackdoorMBTI will expedite future research in backdoor defense methods within a multimodal context. Code is available at https://github.com/SJTUHaiyangYu/BackdoorMBTI.

翻译：过去几年中，后门攻击的出现给深度学习系统带来了重大挑战，攻击者能够在神经网络中植入后门。当带有触发器的数据被后门模型处理时，会导致攻击者预期的错误预测，而正常数据则产生常规结果。后门攻击的范围正从计算机视觉领域扩展到自然语言处理和语音识别等领域。然而，现有的后门防御方法通常针对特定数据模态设计，限制了其在多模态场景中的应用。尽管多模态学习在人脸识别、情感分析、动作识别、视觉问答等领域展现出高度适用性，这些模型的安全性仍是关键问题。具体而言，目前尚不存在针对多模态应用或相关任务的后门基准测试工具。为促进多模态后门领域的研究，我们推出了BackdoorMBTI——首个面向多模态评估的后门学习工具包及基准测试平台，涵盖来自11个常用数据集的三种代表性模态。BackdoorMBTI提供系统化的后门学习流程，包括数据处理、数据投毒、后门训练与评估。生成的投毒数据集和后门模型支持对后门防御策略的精细化评估。鉴于模态的多样性，BackdoorMBTI支持跨不同数据类型的系统性评估。此外，该工具包提供了处理后门学习中实际因素（如数据质量问题与错误标签）的标准化方法。我们预期BackdoorMBTI将加速多模态背景下后门防御方法的未来研究。代码发布于https://github.com/SJTUHaiyangYu/BackdoorMBTI。