Large language models (LLMs) enable state-of-the-art semantic capabilities to be added to software systems such as semantic search of unstructured documents and text generation. However, these models are computationally expensive. At scale, the cost of serving thousands of users increases massively affecting also user experience. To address this problem, semantic caches are used to check for answers to similar queries (that may have been phrased differently) without hitting the LLM service. Due to the nature of these semantic cache techniques that rely on query embeddings, there is a high chance of errors impacting user confidence in the system. Adopting semantic cache techniques usually requires testing the effectiveness of a semantic cache (accurate cache hits and misses) which requires a labelled test set of similar queries and responses which is often unavailable. In this paper, we present VaryGen, an approach for using LLMs for test input generation that produces similar questions from unstructured text documents. Our novel approach uses the reasoning capabilities of LLMs to 1) adapt queries to the domain, 2) synthesise subtle variations to queries, and 3) evaluate the synthesised test dataset. We evaluated our approach in the domain of a student question and answer system by qualitatively analysing 100 generated queries and result pairs, and conducting an empirical case study with an open source semantic cache. Our results show that query pairs satisfy human expectations of similarity and our generated data demonstrates failure cases of a semantic cache. Additionally, we also evaluate our approach on Qasper dataset. This work is an important first step into test input generation for semantic applications and presents considerations for practitioners when calibrating a semantic cache.

翻译：大型语言模型（LLMs）为软件系统增添了最先进的语义能力，例如非结构化文档的语义搜索和文本生成。然而，这些模型计算成本高昂。在大规模应用场景下，服务于数千名用户的成本急剧增加，同时严重影响用户体验。为解决此问题，语义缓存技术被用于检查相似查询（可能以不同措辞表述）的答案，从而避免调用LLM服务。由于这类依赖查询嵌入的语义缓存技术本质，其存在较高出错概率，进而影响用户对系统的信任。采用语义缓存技术通常需要测试缓存有效性（准确命中与未命中），而这需要标注的相似查询及响应测试集——此类数据集往往难以获取。本文提出VaryGen方法，该方法利用LLM从非结构化文本文档生成测试输入以产生相似问题。我们的创新方法借助LLM的推理能力实现：1）将查询适配至特定领域；2）合成查询的细微变体；3）评估所合成的测试数据集。我们通过定性分析100组生成查询与结果对，并基于开源语义缓存开展实证案例研究，在问答系统中验证了该方法。实验结果表明，查询对符合人类对相似性的预期，且生成数据揭示了语义缓存的失败案例。此外，我们还在Qasper数据集上进行了方法评估。该工作是为语义应用生成测试输入的重要初步探索，并为实践者校准语义缓存提供了参考建议。