Despite the advanced capabilities of contemporary machine learning (ML) models, they remain vulnerable to adversarial and backdoor attacks. This vulnerability is particularly concerning in real-world deployments, where compromised models may exhibit unpredictable behavior in critical scenarios. Such risks are heightened by the prevalent practice of collecting massive, internet-sourced datasets for pre-training multimodal models, as these datasets may harbor backdoors. Various techniques have been proposed to mitigate the effects of backdooring in these models such as CleanCLIP which is the current state-of-the-art approach. In this work, we demonstrate that the efficacy of CleanCLIP in mitigating backdoors is highly dependent on the particular objective used during model pre-training. We observe that stronger pre-training objectives correlate with harder to remove backdoors behaviors. We show this by training multimodal models on two large datasets consisting of 3 million (CC3M) and 6 million (CC6M) datapoints, under various pre-training objectives, followed by poison removal using CleanCLIP. We find that CleanCLIP is ineffective when stronger pre-training objectives are used, even with extensive hyperparameter tuning. Our findings underscore critical considerations for ML practitioners who pre-train models using large-scale web-curated data and are concerned about potential backdoor threats. Notably, our results suggest that simpler pre-training objectives are more amenable to effective backdoor removal. This insight is pivotal for practitioners seeking to balance the trade-offs between using stronger pre-training objectives and security against backdoor attacks.

翻译：尽管当代机器学习模型具备先进能力，它们仍易遭受对抗攻击和后门攻击。这种脆弱性在实际部署中尤为令人担忧，因为受损模型可能在关键场景中表现出不可预测的行为。当前普遍采用的大规模网络数据集预训练多模态模型的实践加剧了此类风险，因为这些数据集可能隐藏后门。已有多种技术被提出用于缓解模型中的后门效应，例如CleanCLIP——当前最先进的方法。本研究表明，CleanCLIP在缓解后门方面的有效性高度依赖于模型预训练期间使用的特定目标函数。我们观察到，更强的预训练目标与更难以移除的后门行为之间存在相关性。通过在包含300万（CC3M）和600万（CC6M）数据点的两个大规模数据集上训练多模态模型，采用不同预训练目标，随后使用CleanCLIP进行中毒移除，我们验证了这一点。我们发现，当使用更强的预训练目标时，即使进行广泛的超参数调优，CleanCLIP也效果不佳。我们的发现强调了机器学习从业者在利用大规模网络策展数据预训练模型并关注潜在后门威胁时需要考虑的关键因素。值得注意的是，我们的结果表明，更简单的预训练目标更有利于有效后门移除。这一洞见对于寻求在更优预训练目标与后门攻击安全性之间取得平衡的从业者至关重要。