Text embedding models have significantly contributed to advancements in natural language processing by adeptly capturing semantic properties of textual data. However, the ability of these models to generalize across a wide range of syntactic contexts remains under-explored. In this paper, we first develop an evaluation set, named \textbf{SR}, to scrutinize the capability for syntax understanding of text embedding models from two crucial syntactic aspects: Structural heuristics, and Relational understanding among concepts, as revealed by the performance gaps in previous studies. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges, and such ineffectiveness becomes even more apparent when evaluated against existing benchmark datasets. Furthermore, we conduct rigorous analysis to unearth factors that lead to such limitations and examine why previous evaluations fail to detect such ineffectiveness. Lastly, we propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios. This study serves to highlight the hurdles associated with syntactic generalization and provides pragmatic guidance for boosting model performance across varied syntactic contexts.

翻译：文本嵌入模型通过精确捕捉文本数据的语义特性，为自然语言处理的进步做出了重要贡献。然而，这些模型在广泛句法语境中的泛化能力仍未被充分探索。本文首先构建了一个名为 \textbf{SR} 的评估集，从两个关键句法维度——结构性启发式与概念关系理解——细察文本嵌入模型的句法理解能力，这些维度通过先前研究中的性能差距得以揭示。我们的研究发现表明，现有文本嵌入模型尚未充分解决这些句法理解挑战，且这种不足在现有基准数据集上的评估中表现得更为显著。此外，我们进行了严谨分析以挖掘导致此类局限性的因素，并探究先前评估为何未能检测到这种效能不足。最后，我们提出了增强文本嵌入模型在多样化句法场景中泛化能力的策略。本研究旨在凸显句法泛化所面临的障碍，并为提升模型在不同句法语境下的性能提供实用指导。