Augmented generation techniques such as Retrieval-Augmented Generation (RAG) and Cache-Augmented Generation (CAG) have revolutionized the field by enhancing large language model (LLM) outputs with external knowledge and cached information. However, the integration of vector databases, which serve as a backbone for these augmentations, introduces critical challenges, particularly in ensuring accurate vector matching. False vector matching in these databases can significantly compromise the integrity and reliability of LLM outputs, leading to misinformation or erroneous responses. Despite the crucial impact of these issues, there is a notable research gap in methods to effectively detect and address false vector matches in LLM-augmented generation. This paper presents MeTMaP, a metamorphic testing framework developed to identify false vector matching in LLM-augmented generation systems. We derive eight metamorphic relations (MRs) from six NLP datasets, which form our method's core, based on the idea that semantically similar texts should match and dissimilar ones should not. MeTMaP uses these MRs to create sentence triplets for testing, simulating real-world LLM scenarios. Our evaluation of MeTMaP over 203 vector matching configurations, involving 29 embedding models and 7 distance metrics, uncovers significant inaccuracies. The results, showing a maximum accuracy of only 41.51\% on our tests compared to the original datasets, emphasize the widespread issue of false matches in vector matching methods and the critical need for effective detection and mitigation in LLM-augmented applications.

翻译：摘要：检索增强生成（RAG）与缓存增强生成（CAG）等增强生成技术，通过引入外部知识与缓存信息显著提升了大语言模型（LLM）的输出质量。然而，作为这些增强技术核心支撑的向量数据库的集成应用带来了关键挑战，尤其在确保向量匹配准确性方面。此类数据库中的错误向量匹配会严重损害LLM输出的完整性与可靠性，导致信息误导或错误响应。尽管该问题影响重大，但目前针对LLM增强生成系统中错误向量匹配的有效检测与处理方法仍存在显著研究空白。本文提出MeTMaP——一个专为识别LLM增强生成系统中错误向量匹配而设计的蜕变测试框架。基于"语义相似文本应当匹配、语义相异文本不应匹配"的核心思想，我们从六个自然语言处理数据集中推导出八种蜕变关系（MRs）作为该方法的核心。MeTMaP利用这些MRs构建用于测试的句子三元组，模拟真实LLM应用场景。通过对涵盖29个嵌入模型与7种距离度量的203种向量匹配配置进行评测，我们发现了显著的不准确性。测试结果揭示：相较原始数据集，本方法中最高准确率仅为41.51%，这凸显了向量匹配方法中错误匹配问题的普遍性，以及在LLM增强应用中开展有效检测与缓解工作的紧迫性。