Instruction embedding models have become common among state-of-the-art models, however are evaluated using a single prompt per task. The single-point evaluation ignores a main problem of the instruction-based approach namely: sensitivity to the phrasing of the instruction. We present an empirical study of prompt sensitivity across 6 embedding models, 11 datasets, and 15 task-specific prompts per dataset, a total of 990. We show that reported scores misrepresent the distribution of scores over plausible prompts. The default prompt can both systematically understate or overstate performance. Furthermore, we show that the leaderboard ranking is not robust to prompt selection: by choosing prompts favorably, any model in our study can be promoted to first place. Our findings suggest that single-prompt evaluation is insufficient for instruction-tuned embedding models and that benchmarks should incorporate prompt robustness, either by evaluating over multiple prompts or by reporting sensitivity alongside point estimates.

翻译：指令嵌入模型在最新模型中已变得普遍，然而这些模型在每个任务中仅使用单一提示进行评估。这种单点评估忽略了一个基于指令方法的主要问题，即：对指令措辞的敏感性。我们针对6个嵌入模型、11个数据集以及每个数据集15个任务特定提示（共计990个）进行了提示敏感性的实证研究。研究表明，已报告的分数未能反映在合理提示上得分的分布情况。默认提示可能会系统性地低估或高估性能。此外，我们发现排行榜排名对提示选择不具有鲁棒性：通过选择有利的提示，我们研究中的任何模型都可能被提升至首位。我们的发现表明，对于指令微调的嵌入模型，单提示评估是不够的，基准测试应纳入提示鲁棒性，要么通过多个提示进行评估，要么在点估计的同时报告敏感性指标。