In enterprise search, building high-quality datasets at scale remains a central challenge due to the difficulty of acquiring labeled data. To resolve this challenge, we propose an efficient approach to fine-tune small language models (SLMs) for accurate relevance labeling, enabling high-throughput, domain-specific labeling comparable or even better in quality to that of state-of-the-art large language models (LLMs). To overcome the lack of high-quality and accessible datasets in the enterprise domain, our method leverages on synthetic data generation. Specifically, we employ an LLM to synthesize realistic enterprise queries from a seed document, apply BM25 to retrieve hard negatives, and use a teacher LLM to assign relevance scores. The resulting dataset is then distilled into an SLM, producing a compact relevance labeler. We evaluate our approach on a high-quality benchmark consisting of 923 enterprise query-document pairs annotated by trained human annotators, and show that the distilled SLM achieves agreement with human judgments on par with or better than the teacher LLM. Furthermore, our fine-tuned labeler substantially improves throughput, achieving 17 times increase while also being 19 times more cost-effective. This approach enables scalable and cost-effective relevance labeling for enterprise-scale retrieval applications, supporting rapid offline evaluation and iteration in real-world settings.

翻译：在企业搜索中，由于获取标注数据的困难，大规模构建高质量数据集仍然是一个核心挑战。为解决这一挑战，我们提出了一种高效的方法来微调小型语言模型（SLMs），以实现准确的相关性标注，其标注质量可与最先进的大型语言模型（LLMs）相媲美甚至更优，同时支持高吞吐量和领域特定标注。为克服企业领域缺乏高质量且可访问数据集的问题，我们的方法利用了合成数据生成技术。具体而言，我们使用LLM从种子文档中合成真实的企业查询，应用BM25检索困难负样本，并利用教师LLM分配相关性分数。随后将生成的数据集蒸馏到SLM中，从而产生一个紧凑的相关性标注器。我们在一个由训练有素的人工标注者标注的923对企业查询-文档对组成的高质量基准上评估了我们的方法，结果表明蒸馏后的SLM与人类判断的一致性达到或优于教师LLM。此外，我们微调后的标注器显著提高了吞吐量，实现了17倍的提升，同时成本效益也提高了19倍。该方法为企业级检索应用提供了可扩展且成本效益高的相关性标注方案，支持实际场景中的快速离线评估与迭代。