Large Language Models (LLMs) have emerged as promising zero-shot rankers, but their performance is highly sensitive to prompt formulation. In particular, role-play prompts, where the model is assigned a functional role or identity, often give more robust and accurate relevance rankings. However, the mechanisms and diversity of role-play effects remain underexplored, limiting both effective use and interpretability. In this work, we systematically examine how role-play variations influence zero-shot LLM rankers. We employ causal intervention techniques from mechanistic interpretability to trace how role-play information shapes relevance judgments in LLMs. Our analysis reveals that (1) careful formulation of role descriptions have a large effect on the ranking quality of the LLM; (2) role-play signals are predominantly encoded in early layers and communicate with task instructions in middle layers, while receiving limited interaction with query or document representations. Specifically, we identify a group of attention heads that encode information critical for role-conditioned relevance. These findings not only shed light on the inner workings of role-play in LLM ranking but also offer guidance for designing more effective prompts in IR and beyond, pointing toward broader opportunities for leveraging role-play in zero-shot applications.

翻译：大型语言模型（LLM）已成为具有前景的零样本排序器，但其性能对提示的构建方式高度敏感。特别是角色扮演提示——即赋予模型功能性角色或身份——通常能产生更稳健且准确的相关性排序。然而，角色扮演效应的机制与多样性仍未得到充分探索，这既限制了其有效使用，也影响了模型的可解释性。在本研究中，我们系统性地考察了角色扮演的变体如何影响零样本LLM排序器。我们采用机制可解释性中的因果干预技术，追踪角色扮演信息如何在LLM中塑造相关性判断。我们的分析表明：（1）角色描述的精心构建对LLM的排序质量有显著影响；（2）角色扮演信号主要编码于模型的早期层，并在中间层与任务指令进行交互，而与查询或文档表征的交互有限。具体而言，我们识别出一组注意力头，它们编码了对角色条件相关性判断至关重要的信息。这些发现不仅揭示了角色扮演在LLM排序中的内部工作机制，也为信息检索及其他领域设计更有效的提示提供了指导，同时指出了在零样本应用中更广泛利用角色扮演的潜在机会。