Despite the advanced capabilities of contemporary machine learning (ML) models, they remain vulnerable to adversarial and backdoor attacks. This vulnerability is particularly concerning in real-world deployments, where compromised models may exhibit unpredictable behavior in critical scenarios. Such risks are heightened by the prevalent practice of collecting massive, internet-sourced datasets for pre-training multimodal models, as these datasets may harbor backdoors. Various techniques have been proposed to mitigate the effects of backdooring in these models such as CleanCLIP which is the current state-of-the-art approach. In this work, we demonstrate that the efficacy of CleanCLIP in mitigating backdoors is highly dependent on the particular objective used during model pre-training. We observe that stronger pre-training objectives correlate with harder to remove backdoors behaviors. We show this by training multimodal models on two large datasets consisting of 3 million (CC3M) and 6 million (CC6M) datapoints, under various pre-training objectives, followed by poison removal using CleanCLIP. We find that CleanCLIP is ineffective when stronger pre-training objectives are used, even with extensive hyperparameter tuning. Our findings underscore critical considerations for ML practitioners who pre-train models using large-scale web-curated data and are concerned about potential backdoor threats. Notably, our results suggest that simpler pre-training objectives are more amenable to effective backdoor removal. This insight is pivotal for practitioners seeking to balance the trade-offs between using stronger pre-training objectives and security against backdoor attacks.

翻译：尽管当代机器学习模型具备先进能力，它们仍然易受对抗攻击和后门攻击。这种脆弱性在实际部署中尤为令人担忧，因为受损模型可能在关键场景中表现出不可预测的行为。由于普遍采用收集海量互联网来源数据集进行多模态模型预训练的做法，这些数据集可能隐藏后门，从而加剧了此类风险。目前已提出多种缓解模型后门影响的技术，例如CleanCLIP这一当前最先进的方法。在本研究中，我们证明CleanCLIP在缓解后门方面的有效性高度依赖于模型预训练过程中使用的特定目标。我们观察到，更强的预训练目标与更难移除的后门行为相关。通过在包含300万（CC3M）和600万（CC6M）数据点的两个大型数据集上，采用多种预训练目标训练多模态模型，随后使用CleanCLIP进行毒化移除，我们验证了这一现象。研究发现，当使用更强的预训练目标时，即使进行广泛的超参数调优，CleanCLIP也效果不佳。我们的发现强调了使用大规模网络整理数据进行预训练的机器学习从业者需要关注的关键考量。值得注意的是，我们的结果表明，更简单的预训练目标更有利于有效移除后门。这一见解对于寻求权衡使用更强预训练目标与防范后门攻击安全性的从业者至关重要。