Vector search and retrieval-augmented generation (RAG) rest on the assumption that cosine similarity between text embeddings reflects conceptual relatedness. We measure where this assumption breaks. We build an augmented citation graph over 3.58M scientific papers and partition it via Leiden CPM at two granularities: sub-field (L1) and research-agenda (L2, hierarchical inside each L1). Four state-of-the-art embeddings (Gemini, Qwen3-8B, Qwen3-0.6B, SPECTER2) clear the L1 bar reasonably (45-52% top-10 same-rate) but stop working at L2: only 15-21% of top-10 neighbors share the query's research agenda. In absolute terms, 8 of every 10 retrieved papers are off-agenda. The failure is universal across eight scientific domains and all four models; SPECTER2, despite its citation-based contrastive training, is the weakest. As a diagnostic probe, we test whether the same augmented graph also functions as a retrieval signal: a deliberately simple citation-count rerank reaches 57.7% top-1 L2 on top of LLM-expanded Boolean retrieval and 59.6% on top of plain BM25, on 80 curated agenda queries -- about 9 points above the best cosine retriever (Gemini, 50.6%) and 20 points above BM25 alone (39.3%). The probe isolates a slice of the agenda-matching signal the graph carries but the embeddings miss, connecting recent theoretical limits on single-vector retrieval to a concrete failure mode of scientific RAG.
翻译:向量搜索与检索增强生成(RAG)依赖于文本嵌入的余弦相似度能反映概念相关性的假设。我们度量了这一假设失效的情形。我们构建了包含358万篇科学论文的增广引文图,并通过Leiden CPM将其分为两个粒度层级:子领域(L1)和研究议程(L2,在每个L1内部层级化划分)。四种最先进的嵌入模型(Gemini、Qwen3-8B、Qwen3-0.6B、SPECTER2)在L1层级表现尚可(前10邻居相同率为45-52%),但在L2层级失效:仅15-21%的前10邻居与查询共享相同的研究议程。从绝对数值来看,每检索10篇论文就有8篇偏离原定议程。这一失败在八个科学领域和全部四个模型中普遍存在;其中SPECTER2尽管采用了基于引文的对比训练,表现却最弱。作为诊断性探针,我们检验了同一增广引文图是否也能充当检索信号:在80个精心设计的议程查询上,将简单的引文计数重排序叠加于大语言模型扩展的布尔检索之上,可达到57.7%的L2首篇命中率;叠加于纯BM25之上则达59.6%——分别比最佳余弦检索器(Gemini,50.6%)高出约9个百分点,比纯BM25(39.3%)高出20个百分点。该探针从引文图所携带但嵌入模型遗漏的议程匹配信号中分离出一个切片,将单向量检索的近期理论极限与科学RAG的具体失效模式联系起来。