We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights. With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance. Our evaluation of 16 popular embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data retrieval capabilities. Our work identifies unique challenges in conversational data retrieval (implicit state recognition, turn dynamics, contextual references) while providing practical query templates and detailed error analysis across different task categories. The benchmark dataset and code are available at https://github.com/l-yohai/CDR-Benchmark.

翻译：我们提出了对话数据检索（CDR）基准，这是首个用于评估为产品洞察检索对话数据系统的综合性测试集。该基准包含跨越五项分析任务的1.6k个查询和9.1k段对话，为衡量对话数据检索性能提供了可靠标准。我们对16种流行嵌入模型的评估表明，即使最佳模型在NDCG@10指标上也仅能达到约0.51，这揭示了文档数据检索与对话数据检索能力之间存在显著差距。本研究明确了对话数据检索面临的独特挑战（隐含状态识别、话轮动态、上下文指代），同时提供了实用的查询模板及跨不同任务类别的详细错误分析。基准数据集与代码已发布于 https://github.com/l-yohai/CDR-Benchmark。