The increasing depth of parametric domain knowledge in large language models (LLMs) is fueling their rapid deployment in real-world applications. In high-stakes and knowledge-intensive tasks, understanding model vulnerabilities is essential for quantifying the trustworthiness of model predictions and regulating their use. The recent discovery of named entities as adversarial examples in natural language processing tasks raises questions about their potential guises in other settings. Here, we propose a powerscaled distance-weighted sampling scheme in embedding space to discover diverse adversarial entities as distractors. We demonstrate its advantage over random sampling in adversarial question answering on biomedical topics. Our approach enables the exploration of different regions on the attack surface, which reveals two regimes of adversarial entities that markedly differ in their characteristics. Moreover, we show that the attacks successfully manipulate token-wise Shapley value explanations, which become deceptive in the adversarial setting. Our investigations illustrate the brittleness of domain knowledge in LLMs and reveal a shortcoming of standard evaluations for high-capacity models.

翻译：大语言模型中参数化领域知识的日益加深正在推动其在现实世界应用中的快速部署。在高风险、知识密集型任务中，理解模型脆弱性对于量化模型预测的可信度并规范其使用至关重要。近期在自然语言处理任务中发现命名实体可作为对抗性样本，这一发现引发了对这些实体在其他应用场景中潜在表现形式的思考。本文提出一种基于幂律缩放距离加权采样的嵌入空间采样方案，用于发现多样化的对抗性实体作为干扰项。我们证明了该方法在生物医学主题对抗性问答中相比随机采样的优越性。该方案能够探索攻击表面的不同区域，揭示出两类特征截然不同的对抗性实体。此外，研究表明这些攻击成功操纵了基于token的沙普利值解释，使其在对抗性设置中具有欺骗性。我们的研究揭示了大型语言模型中领域知识的脆弱性，并指出了针对高容量模型的标准评估方法的缺陷。