We designed a Retrieval-Augmented Generation (RAG) system to provide large language models with relevant documents for answering domain-specific questions about Pittsburgh and Carnegie Mellon University (CMU). We extracted over 1,800 subpages using a greedy scraping strategy and employed a hybrid annotation process, combining manual and Mistral-generated question-answer pairs, achieving an inter-annotator agreement (IAA) score of 0.7625. Our RAG framework integrates BM25 and FAISS retrievers, enhanced with a reranker for improved document retrieval accuracy. Experimental results show that the RAG system significantly outperforms a non-RAG baseline, particularly in time-sensitive and complex queries, with an F1 score improvement from 5.45% to 42.21% and recall of 56.18%. This study demonstrates the potential of RAG systems in enhancing answer precision and relevance, while identifying areas for further optimization in document retrieval and model training.

翻译：我们设计了一个检索增强生成系统，旨在为大型语言模型提供相关文档，以回答关于匹兹堡和卡内基梅隆大学的领域特定问题。我们采用贪心爬取策略提取了超过1800个子页面，并采用混合标注流程，结合人工与Mistral生成的问题-答案对，实现了0.7625的标注者间一致性分数。我们的RAG框架集成了BM25和FAISS检索器，并通过重排序器增强以提升文档检索精度。实验结果表明，该RAG系统显著优于非RAG基线，尤其在时间敏感和复杂查询方面，其F1分数从5.45%提升至42.21%，召回率达到56.18%。本研究证明了RAG系统在提升答案精确度和相关性方面的潜力，同时指出了文档检索和模型训练方面有待进一步优化的领域。