Embedding Models for Supervised Automatic Extraction and Classification of Named Entities in Scientific Acknowledgements

from arxiv, The present paper is an extended version of the article Evaluation of Embedding Models for Automatic Extraction and Classification of Acknowledged Entities in Scientific Documents (Smirnova and Mayr, 2022) presented at the 3rd Workshop on Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE2022). arXiv admin note: substantial text overlap with arXiv:2206.10939

Acknowledgments in scientific papers may give an insight into aspects of the scientific community, such as reward systems, collaboration patterns, and hidden research trends. The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities from the acknowledgment text in scientific papers. We trained and implemented a named entity recognition (NER) task using the Flair NLP framework. The training was conducted using three default Flair NER models with four differently-sized corpora and different versions of the Flair NLP framework. The Flair Embeddings model trained on the medium corpus with the latest FLAIR version showed the best accuracy of 0.79. Expanding the size of a training corpus from very small to medium size massively increased the accuracy of all training algorithms, but further expansion of the training corpus did not bring further improvement. Moreover, the performance of the model slightly deteriorated. Our model is able to recognize six entity types: funding agency, grant number, individuals, university, corporation, and miscellaneous. The model works more precisely for some entity types than for others; thus, individuals and grant numbers showed a very good F1-Score over 0.9. Most of the previous works on acknowledgment analysis were limited by the manual evaluation of data and therefore by the amount of processed data. This model can be applied for the comprehensive analysis of acknowledgment texts and may potentially make a great contribution to the field of automated acknowledgment analysis.

翻译：科学论文中的致谢可能揭示科学共同体的若干方面，如奖励机制、合作模式及隐性研究趋势。本文旨在评估不同嵌入模型在科学论文致谢文本中自动抽取与分类致谢实体的性能表现。我们采用Flair NLP框架训练并实现了命名实体识别（NER）任务。训练过程中使用了三种默认Flair NER模型，结合四种不同规模的语料库与不同版本的Flair NLP框架。基于中等规模语料库并采用最新Flair版本的Flair嵌入模型取得了0.79的最佳准确率。将训练语料库规模从极小扩展至中等可显著提升所有训练算法的准确率，但进一步扩大语料库规模并未带来额外提升，反而导致模型性能略有下降。我们的模型能够识别六类实体：资助机构、项目编号、个人、大学、企业及其他杂项实体。模型对不同实体类型的识别精度存在差异，其中个人与项目编号的F1得分超过0.9，表现优异。此前大多数致谢分析研究受限于人工数据评估，进而制约了可处理数据的规模。本模型可应用于致谢文本的全面分析，有望对自动化致谢分析领域作出重要贡献。