Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token. Building on these insights, we introduce Dyslexify - a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, dyslexify improves performance by up to 22.06% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.

翻译：排版攻击通过向图像中注入文本来利用多模态系统，导致定向错误分类、恶意内容生成甚至视觉语言模型越狱。本研究分析了CLIP视觉编码器在排版攻击下的行为机制，定位到模型后半部分层中存在专门提取排版信息并将其因果传递至cls标记的特定注意力头。基于这些发现，我们提出Dyslexify——一种通过选择性消融由注意力头构成的排版回路来防御CLIP模型免受排版攻击的方法。该方法无需微调，在ImageNet-100的排版变体数据集上性能提升最高达22.06%，同时标准ImageNet-100准确率下降不足1%，并在皮肤病变诊断的医学基础模型中验证了其有效性。值得注意的是，这种免训练方法与当前依赖微调的最先进排版防御技术相比仍具竞争力。为此，我们发布了一系列对排版攻击具有显著鲁棒性的Dyslexic CLIP模型。这些模型可作为即插即用的替代方案，广泛应用于文本识别效用低于文本操纵风险的安全关键领域。