Current sentence embedding evaluations typically rely on static test beds like the Massive Text Embedding Benchmark (MTEB). While invaluable, repeated tuning on a fixed suite can inflate reported scores and obscure real-world robustness. We introduce the Paraphrasing Text Embedding Benchmark (PTEB), a dynamic protocol that stochastically generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. Using a cost-efficient LLM-based method grounded in gold ratings and human validation, we show that LLMs generate token-diverse but semantically preserving paraphrases. Across 7 MTEB tasks, we validate our hypothesis that the performance of sentence encoders is sensitive to changes in token space even when semantics remain fixed. We also observe that smaller models are not disproportionately affected relative to larger ones. Our results are statistically robust over multiple runs spanning 20 datasets and 25 languages. More generally, we aim to propose a new evaluation paradigm in NLP that relies less on static, pre-defined benchmarks but shifts towards dynamic, stochastic evaluation leveraging eval-time compute.

翻译：当前的句子嵌入评估通常依赖于静态测试集，如大规模文本嵌入基准（MTEB）。虽然这些基准极具价值，但在固定套件上反复调优可能导致报告分数虚高，并掩盖模型在真实世界中的鲁棒性。我们引入了复述文本嵌入基准（PTEB），这是一种动态评估协议，其在评估时随机生成保持语义的复述，并聚合多次运行的结果。我们采用一种基于大语言模型、成本效益高且基于黄金评分和人工验证的方法，证明了大语言模型能够生成词汇多样但语义保持不变的复述。在7个MTEB任务上的实验验证了我们的假设：即使语义保持不变，句子编码器的性能也对词汇空间的变化敏感。我们还观察到，较小模型相对于较大模型并未受到不成比例的影响。我们的结果在涵盖20个数据集和25种语言的多次运行中具有统计鲁棒性。更广泛而言，我们的目标是提出一种自然语言处理领域的新评估范式，该范式减少对静态、预定义基准的依赖，转而利用评估时的计算资源，向动态、随机的评估方式转变。