This work considers a black-box threat model in which adversaries attempt to propagate arbitrary non-relevant content in search. We show that retrievers, rerankers, and LLM relevance judges are all highly vulnerable to attacks that enable arbitrary content to be promoted to the top of search results and to be assigned perfect relevance scores. We investigate how attackers may achieve this via content injection, injecting arbitrary sentences into relevant passages or query terms into arbitrary passages. Our study analyzes how factors such as model class and size, the balance between relevant and non-relevant content, injection location, toxicity and severity of injected content, and the role of LLM-generated content influence attack success, yielding novel, concerning, and often counterintuitive results. Our results reveal a weakness in embedding models, LLM-based scoring models, and generative LLMs, raising concerns about the general robustness, safety, and trustworthiness of language models regardless of the type of model or the role in which they are employed. We also emphasize the challenges of robust defenses against these attacks. Classifiers and more carefully prompted LLM judges often fail to recognize passages with content injection, especially when considering diverse text topics and styles. Our findings highlight the need for further research into arbitrary content injection attacks. We release our code for further study.

翻译：本研究探讨一种黑盒威胁模型，其中攻击者试图在搜索中传播任意不相关内容。我们证明，检索器、重排序器和基于LLM的相关性评估器均极易受到攻击，导致任意内容被推至搜索结果顶部并获得完美相关性评分。我们研究攻击者如何通过内容注入实现此目的——将任意句子注入相关段落或将查询词注入任意段落。本研究系统分析了模型类别与规模、相关与不相关内容之间的平衡、注入位置、注入内容的毒性及严重程度、LLM生成内容的作用等因素如何影响攻击成功率，并得出新颖、令人担忧且常违反直觉的结果。我们的研究结果揭示了嵌入模型、基于LLM的评分模型以及生成式LLM的共性弱点，引发了对语言模型整体鲁棒性、安全性与可信度的普遍担忧——无论模型类型或其应用场景为何。我们同时强调了针对此类攻击构建稳健防御机制所面临的挑战：分类器与经过更精细提示的LLM评估器往往无法识别存在内容注入的段落，尤其在涉及多样化文本主题与风格时。本研究结果凸显了对任意内容注入攻击开展进一步研究的迫切需求。我们已公开实验代码以供后续研究。