Research profiles highlight scientists' research focus, enabling talent discovery and collaborations, but are often outdated. Automated, scalable methods are urgently needed to keep profiles current. We design and evaluate two Large Language Models (LLMs)-based methods to generate scientific interest profiles--one summarizing PubMed abstracts and the other using Medical Subject Headings (MeSH) terms--comparing them with researchers' self-summarized interests. We collected titles, MeSH terms, and abstracts of PubMed publications for 595 faculty at Columbia University Irving Medical Center, obtaining human-written profiles for 167. GPT-4o-mini was prompted to summarize each researcher's interests. Manual and automated evaluations characterized similarities between machine-generated and self-written profiles. The similarity study showed low ROUGE-L, BLEU, and METEOR scores, reflecting little terminological overlap. BERTScore analysis revealed moderate semantic similarity (F1: 0.542 for MeSH-based, 0.555 for abstract-based), despite low lexical overlap. In validation, paraphrased summaries achieved a higher F1 of 0.851. Comparing original and manually paraphrased summaries indicated limitations of such metrics. Kullback-Leibler (KL) Divergence of TF-IDF values (8.56 for MeSH-based, 8.58 for abstract-based) suggests machine summaries employ different keywords than human-written ones. Manual reviews showed 77.78% rated MeSH-based profiling "good" or "excellent," with readability rated favorably in 93.44% of cases, though granularity and accuracy varied. Panel reviews favored 67.86% of MeSH-derived profiles over abstract-derived ones. LLMs promise to automate scientific interest profiling at scale. MeSH-derived profiles have better readability than abstract-derived ones. Machine-generated summaries differ from human-written ones in concept choice, with the latter initiating more novel ideas.

翻译：研究画像能够凸显科学家的研究重点，促进人才发现与合作，但往往存在信息过时的问题。当前亟需自动化、可扩展的方法来保持画像的时效性。我们设计并评估了两种基于大语言模型（LLMs）生成科学兴趣画像的方法——一种总结PubMed摘要，另一种利用医学主题词（MeSH）——并将其与研究者的自我总结兴趣进行比较。我们收集了哥伦比亚大学欧文医学中心595位教职人员的PubMed出版物标题、MeSH术语和摘要，并获取了其中167位的人工撰写画像。我们使用GPT-4o-mini来总结每位研究者的兴趣。通过人工和自动评估，刻画了机器生成画像与自我撰写画像之间的相似性。相似性研究显示，ROUGE-L、BLEU和METEOR得分较低，反映出术语重叠度很小。BERTScore分析揭示了中等的语义相似性（基于MeSH的F1：0.542，基于摘要的F1：0.555），尽管词汇重叠度低。在验证中，经过改写的总结获得了更高的F1值0.851。比较原始总结与人工改写总结，指出了此类指标的局限性。TF-IDF值的Kullback-Leibler（KL）散度（基于MeSH的为8.56，基于摘要的为8.58）表明机器总结使用了与人工撰写总结不同的关键词。人工评审显示，77.78%的评审者认为基于MeSH的画像"好"或"优秀"，93.44%的案例中可读性评价良好，尽管其精细度和准确性存在差异。小组评审中，67.86%的评审者更青睐基于MeSH生成的画像，而非基于摘要生成的画像。LLMs有望实现大规模自动化科学兴趣画像构建。基于MeSH的画像比基于摘要的画像具有更好的可读性。机器生成的总结在概念选择上与人工撰写的总结存在差异，后者往往能引发更多新颖的想法。