To handle the vast amounts of qualitative data produced in corporate climate communication, stakeholders increasingly rely on Retrieval Augmented Generation (RAG) systems. However, a significant gap remains in evaluating domain-specific information retrieval - the basis for answer generation. To address this challenge, this work simulates the typical tasks of a sustainability analyst by examining 30 sustainability reports with 16 detailed climate-related questions. As a result, we obtain a dataset with over 8.5K unique question-source-answer pairs labeled by different levels of relevance. Furthermore, we develop a use case with the dataset to investigate the integration of expert knowledge into information retrieval with embeddings. Although we show that incorporating expert knowledge works, we also outline the critical limitations of embeddings in knowledge-intensive downstream domains like climate change communication.

翻译：为应对企业气候沟通中产生的大量定性数据，利益相关方日益依赖检索增强生成（RAG）系统。然而，在评估领域特定信息检索（即答案生成的基础）方面仍存在显著差距。为应对这一挑战，本研究通过审阅30份可持续发展报告并提出16个详细的气候相关问题，模拟了可持续发展分析师的典型工作任务。由此我们构建了一个包含超过8500个独特问题-来源-答案对的数据集，并标注了不同级别的相关性。此外，我们利用该数据集开发了应用案例，研究如何将专家知识通过嵌入技术整合到信息检索中。尽管我们证明了融入专家知识的有效性，但也指出了嵌入方法在气候变化沟通等高知识密度下游领域的关键局限性。