In this study, we address one of the challenges of developing NER models for scholarly domains, namely the scarcity of suitable labeled data. We experiment with an approach using predictions from a fine-tuned LLM model to aid non-domain experts in annotating scientific entities within astronomy literature, with the goal of uncovering whether such a collaborative process can approximate domain expertise. Our results reveal moderate agreement between a domain expert and the LLM-assisted non-experts, as well as fair agreement between the domain expert and the LLM model's predictions. In an additional experiment, we compare the performance of finetuned and default LLMs on this task. We have also introduced a specialized scientific entity annotation scheme for astronomy, validated by a domain expert. Our approach adopts a scholarly research contribution-centric perspective, focusing exclusively on scientific entities relevant to the research theme. The resultant dataset, containing 5,000 annotated astronomy article titles, is made publicly available.

翻译：本研究探讨了学术领域NER模型开发中的一项挑战，即合适标注数据的稀缺性。我们尝试利用微调LLM模型的预测结果，辅助非领域专家对天文学文献中的科学实体进行标注，旨在揭示这种协作过程是否能够接近领域专家的专业水平。实验结果显示，领域专家与LLM辅助的非专家之间存在中等一致性，而领域专家与LLM模型预测之间呈现一般一致性。在附加实验中，我们比较了微调与默认LLM在该任务上的性能表现。同时，我们提出了专用于天文学的科学实体标注方案，并经领域专家验证。该方法采用研究贡献导向视角，仅聚焦于与研究主题相关的科学实体。最终构建的数据集包含5,000篇标注后的天文学论文标题，现已公开提供。