Phishing attacks pose a significant threat to Internet users, with cybercriminals elaborately replicating the visual appearance of legitimate websites to deceive victims. Visual similarity-based detection systems have emerged as an effective countermeasure, but their effectiveness and robustness in real-world scenarios have been underexplored. In this paper, we comprehensively scrutinize and evaluate the effectiveness and robustness of popular visual similarity-based anti-phishing models using a large-scale dataset of 451k real-world phishing websites. Our analyses of the effectiveness reveal that while certain visual similarity-based models achieve high accuracy on curated datasets in the experimental settings, they exhibit notably low performance on real-world datasets, highlighting the importance of real-world evaluation. Furthermore, we find that the attackers evade the detectors mainly in three ways: (1) directly attacking the model pipelines, (2) mimicking benign logos, and (3) employing relatively simple strategies such as eliminating logos from screenshots. To statistically assess the resilience and robustness of existing models against adversarial attacks, we categorize the strategies attackers employ into visible and perturbation-based manipulations and apply them to website logos. We then evaluate the models' robustness using these adversarial samples. Our findings reveal potential vulnerabilities in several models, emphasizing the need for more robust visual similarity techniques capable of withstanding sophisticated evasion attempts. We provide actionable insights for enhancing the security of phishing defense systems, encouraging proactive actions.

翻译：钓鱼攻击对互联网用户构成重大威胁，网络犯罪分子通过精心仿冒合法网站的视觉外观来欺骗受害者。基于视觉相似性的检测系统已成为一种有效的应对措施，但其在真实场景中的有效性和鲁棒性尚未得到充分研究。本文利用包含45.1万个真实钓鱼网站的大规模数据集，对主流的基于视觉相似性的反钓鱼模型进行了全面审视与评估。有效性分析表明：尽管某些基于视觉相似性的模型在实验环境的精选数据集上能达到较高准确率，但在真实数据集上表现显著偏低，这凸显了真实场景评估的重要性。进一步研究发现，攻击者主要通过三种方式规避检测：(1) 直接攻击模型处理流程，(2) 仿冒良性标识，(3) 采用相对简单的策略（例如从截图中移除标识）。为系统评估现有模型对抗对抗攻击的恢复能力与鲁棒性，我们将攻击者采用的策略归类为可见性操作和基于扰动的操作，并将其应用于网站标识。随后使用这些对抗样本评估模型的鲁棒性。研究结果揭示了若干模型存在的潜在脆弱性，强调需要开发能够抵御复杂规避手段的、更具鲁棒性的视觉相似性技术。本文为提升钓鱼防御系统的安全性提供了可操作的见解，以促进前瞻性防护措施的制定。