Although indicators based on scholarly citations are widely used to support the evaluation of academic journals, alternatives are needed for scholarly book acquisitions. This article assesses the value of research quality scores from ChatGPT 4o-mini for 9,830 social sciences, arts, and humanities books from 2019 indexed in Scopus, based on their titles and descriptions but not their full texts. Although most books scored the same (3* on a 1* to 4* scale), the citation rates correlate positively but weakly with ChatGPT 4o-mini research quality scores in both the social sciences and the arts and humanities. Part of the reason for the differences was the inclusion of textbooks, short books, and edited collections, all of which tended to be less cited and lower scoring. Some topics also tend to attract many/few citations and/or high/low ChatGPT scores. Descriptions explicitly mentioning theory and/or some methods also associated with higher scores and more citations. Overall, the results provide some evidence that both ChatGPT scores and citation counts are weak indicators of the research quality of books. Whilst not strong enough to support individual book quality judgements, they may help academic librarians seeking to evaluate new book collections, series, or publishers for potential acquisition.
翻译:尽管基于学术引用的指标被广泛用于支持学术期刊的评价,但学术图书采购需要替代性评估方法。本文基于Scopus收录的2019年出版的9,830种社会科学、艺术与人文学科书籍的标题和描述(非全文),评估了ChatGPT 4o-mini生成的研究质量评分的价值。虽然大多数书籍获得相同评分(在1*至4*量表中均为3*),但在社会科学领域以及艺术与人文学科领域,引用率与ChatGPT 4o-mini研究质量评分均呈现微弱正相关。差异的部分原因在于收录了教科书、短篇著作和编辑文集,这些类型往往引用率较低且评分偏低。某些主题也倾向于吸引较多/较少引用和/或较高/较低的ChatGPT评分。明确提及理论和/或某些方法的描述同样与较高评分及较多引用相关。总体而言,结果证明ChatGPT评分和引用计数均属于书籍研究质量的弱指标。虽然不足以支持个体图书的质量判断,但可能有助于学术图书馆员评估待采购的新书合集、丛书或出版商。