Tasks in the aerospace industry heavily rely on searching and reusing large volumes of technical documents, yet there is no public information retrieval (IR) benchmark that reflects the terminology- and query-intent characteristics of this domain. To address this gap, this paper proposes the STELLA (Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information Retrieval Benchmark) framework. Using this framework, we introduce the STELLA benchmark, an aerospace-specific IR evaluation set constructed from NASA Technical Reports Server (NTRS) documents via a systematic pipeline that comprises document layout detection, passage chunking, terminology dictionary construction, synthetic query generation, and cross-lingual extension. The framework generates two types of queries: the Terminology Concordant Query (TCQ), which includes the terminology verbatim to evaluate lexical matching, and the Terminology Agnostic Query (TAQ), which utilizes the terminology's description to assess semantic matching. This enables a disentangled evaluation of the lexical and semantic matching capabilities of embedding models. In addition, we combine Chain-of-Density (CoD) and the Self-Reflection method with query generation to improve quality and implement a hybrid cross-lingual extension that reflects real user querying practices. Evaluation of seven embedding models on the STELLA benchmark shows that large decoder-based embedding models exhibit the strongest semantic understanding, while lexical matching methods such as BM25 remain highly competitive in domains where exact lexical matching technical term is crucial. The STELLA benchmark provides a reproducible foundation for reliable performance evaluation and improvement of embedding models in aerospace-domain IR tasks. The STELLA benchmark can be found in https://huggingface.co/datasets/telepix/STELLA.

翻译：航空航天领域的任务严重依赖于搜索和重用大量技术文档，然而目前尚无公开的信息检索（IR）基准能够反映该领域特有的术语和查询意图特征。为填补这一空白，本文提出了STELLA（构建航空航天信息检索基准的自反思术语感知框架）。利用该框架，我们引入了STELLA基准——一个针对航空航天领域的信息检索评估数据集，该数据集通过系统化的流程从NASA技术报告服务器（NTRS）文档构建而成，流程包括文档布局检测、段落分块、术语词典构建、合成查询生成以及跨语言扩展。该框架生成两种类型的查询：术语一致查询（TCQ），其直接包含术语以评估词汇匹配能力；以及术语无关查询（TAQ），其利用术语的描述来评估语义匹配能力。这使得能够对嵌入模型的词汇匹配和语义匹配能力进行解耦评估。此外，我们将密度链（CoD）和自反思方法与查询生成相结合以提高质量，并实施了反映真实用户查询实践的混合跨语言扩展。在STELLA基准上对七种嵌入模型的评估表明，基于大型解码器的嵌入模型展现出最强的语义理解能力，而在精确匹配技术术语至关重要的领域，如BM25等词汇匹配方法仍然极具竞争力。STELLA基准为航空航天领域信息检索任务中嵌入模型的可靠性能评估与改进提供了可复现的基础。STELLA基准可在 https://huggingface.co/datasets/telepix/STELLA 获取。