The known stylistic biases in LLM judges, such as a preference for verbosity or specific sentence structures, present an underexplored security vulnerability. In this work, we introduce BITE (BIas exploraTion and Exploitation), a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge and artificially inflate the scores it assigns. We cast the selection of stylistic edits as a contextual bandit problem and use a LinUCB policy to adaptively choose edits that maximize the judge's score without access to model parameters or gradients. Empirically, we test BITE across a diverse range of LLM judges and tasks, including both pointwise and pairwise comparisons on chatbot leaderboards and AI-reviewer benchmarks. BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale, all while preserving semantic equivalence. We further assess the attack's stealthiness, showing that BITE evades standard style-control methods and several detection baselines. Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation. Our code is available at https://github.com/xianglinyang/llm-as-a-judge-attack.

翻译：已知的LLM评审者中的风格偏见（如偏好冗长或特定句式）构成了一个尚未充分探索的安全漏洞。本文提出BITE（偏差探索与利用框架），一种黑盒对抗性方法，通过学习保持语义不变的编辑方式，误导LLM评审者以人为抬高其评分。我们将风格编辑的选择建模为上下文赌博机问题，并采用LinUCB策略自适应选择能最大化评审者评分的编辑方式，全程无需访问模型参数或梯度。实验表明，BITE在多种LLM评审者和任务中均有效，涵盖聊天机器人排行榜的逐点与成对比较以及AI审稿人基准测试。BITE的攻击成功率超过65%，在9分制评分中提升1-2分，同时保持语义等价。我们进一步评估了攻击的隐蔽性，证明BITE能规避标准风格控制方法和多种检测基线。本研究揭示了"LLM作为评审者"范式的根本性弱点，并推动了鲁棒且具备攻击感知能力的评估方法发展。我们的代码已开源至https://github.com/xianglinyang/llm-as-a-judge-attack。