Neural Ranking Models (NRMs) are central to modern information retrieval but remain highly vulnerable to adversarial manipulation. Existing attacks often rely on heuristics or surrogate models, limiting effectiveness and transferability. We propose CRAFT, a supervised framework for black-box adversarial rank attacks powered by large language models (LLMs). CRAFT operates in three stages: adversarial dataset generation via retrieval-augmented generation and self-refinement, supervised fine-tuning on curated adversarial examples, and preference-guided optimization to align generations with rank-promotion objectives. Extensive experiments on the MS MARCO passage dataset, TREC Deep Learning 2019, and TREC Deep Learning 2020 benchmarks show that CRAFT significantly outperforms state-of-the-art baselines, achieving higher promotion rates and rank boosts while preserving fluency and semantic fidelity. Moreover, CRAFT transfers effectively across diverse ranking architectures, including cross-encoder, embedding-based, and LLM-based rankers, underscoring vulnerabilities in real-world retrieval systems. This work provides a principled framework for studying adversarial threats in NRMs, underscores the risks of generative AI in rank manipulation, and provides a foundation for developing more robust retrieval systems. To support reproducibility, we publicly release our source code, trained models, and prompt templates.

翻译：神经排序模型（NRMs）是现代信息检索的核心，但依然高度易受对抗性操纵的影响。现有攻击通常依赖启发式方法或代理模型，限制了效果和迁移性。我们提出CRAFT，一种基于大型语言模型（LLMs）的黑盒对抗排序攻击监督式框架。CRAFT通过三个阶段运行：基于检索增强生成与自我精炼的对抗数据集生成、针对精选对抗样本的监督微调、以及偏好引导优化使生成结果对齐排序提升目标。在MS MARCO段落数据集、TREC Deep Learning 2019和TREC Deep Learning 2020基准上的大量实验表明，CRAFT显著优于现有最优基线方法，在保持流畅性和语义保真度的同时实现了更高的提升率和排序增益。此外，CRAFT可有效迁移至多种排序架构，包括交叉编码器、基于嵌入和基于LLM的排序器，揭示了现实检索系统中的脆弱性。本研究为NRM中对抗性威胁研究提供了原理性框架，凸显了生成式AI在排序操纵中的风险，并为构建更鲁棒的检索系统奠定了基础。为支持可复现性，我们公开发布了源代码、训练模型和提示模板。