Large Language Models (LLMs) such as ChatGPT can estimate research quality from article titles and abstracts, and decision makers may increasingly use LLM scores to support research assessment. In this context, it is a concern that LLM scores may be influenced by linguistic signals rather than the quality of the research described. This study investigates whether explicit novelty claims in research abstracts are associated with higher ChatGPT scores. Using 99,245 journal articles submitted to the UK Research Excellence Framework (REF) 2021, ChatGPT scores and percentile rank gains relative to REF departmental average scores (as a proxy for article-level scores) were compared between abstracts with and without any of 282 three-word phrases identified as indicating a novelty claim. Abstracts containing novelty claim phrases had higher mean ChatGPT quality scores in all four main REF research groupings (panels). Moreover, the score difference was higher for the originality dimension of research quality than for significance and rigour in 30 of the 34 broadly field-based Units of Assessment (UoAs) used to categorise research in the REF. Thus, explicit novelty claims probably associate with higher quality scores primarily because they associate with higher scores for the originality dimension. Follow-up tests suggested that abstracts containing novelty claims tended to rank more favourably under ChatGPT than under the REF departmental quality proxy. Whilst the findings suggest that ChatGPT may be influenced by explicit novelty claims, they do not conclusively show that novelty phrases caused the higher scores because the scores and phrases may at least partially reflect more novel research.

翻译：暂无翻译