Generative AI tools exemplified by ChatGPT are becoming a new reality. This study is motivated by the premise that ``AI generated content may exhibit a distinctive behavior that can be separated from scientific articles''. In this study, we show how articles can be generated using means of prompt engineering for various diseases and conditions. We then show how we tested this premise in two phases and prove its validity. Subsequently, we introduce xFakeSci, a novel learning algorithm, that is capable of distinguishing ChatGPT-generated articles from publications produced by scientists. The algorithm is trained using network models driven from both sources. As for the classification step, it was performed using 300 articles per condition. The actual label steps took place against an equal mix of 50 generated articles and 50 authentic PubMed abstracts. The testing also spanned publication periods from 2010 to 2024 and encompassed research on three distinct diseases: cancer, depression, and Alzheimer's. Further, we evaluated the accuracy of the xFakeSci algorithm against some of the classical data mining algorithms (e.g., Support Vector Machines, Regression, and Naive Bayes). The xFakeSci algorithm achieved F1 scores ranging from 80% to 94%, outperforming common data mining algorithms, which scored F1 values between 38% and 52%. We attribute the noticeable difference to the introduction of calibration and a proximity distance heuristic, which underscores this promising performance. Indeed, the prediction of fake science generated by ChatGPT presents a considerable challenge. Nonetheless, the introduction of the xFakeSci algorithm is a significant step on the way to combating fake science.

翻译：以ChatGPT为代表的生成式AI工具正成为新现实。本研究基于"AI生成内容可能表现出与科学文章不同的独特行为"这一前提展开。我们首先展示了如何通过提示工程为多种疾病和病症生成文章，随后分两个阶段验证该前提并证实其有效性。在此基础上，我们提出了一种新型学习算法xFakeSci，该算法能够区分ChatGPT生成的文章与科学家撰写的出版物。算法训练采用来自两类来源的网络模型驱动。分类步骤中，每种病症使用300篇文章；实际标注步骤则针对50篇生成文章与50篇PubMed真实摘要的均衡混合数据。测试还涵盖2010年至2024年的发表周期，并包含癌症、抑郁症和阿尔茨海默病三种不同疾病的研究。此外，我们将xFakeSci算法与部分经典数据挖掘算法（如支持向量机、回归分析和朴素贝叶斯）进行了准确率对比。xFakeSci算法的F1分数达到80%至94%，显著优于常见数据挖掘算法（其F1值在38%至52%之间）。我们将这一显著差异归因于校准机制和邻近距离启发式方法的引入，这凸显了该算法的优异性能。诚然，由ChatGPT生成的虚假科学预测仍是一项重大挑战，但xFakeSci算法的提出为打击虚假科学迈出了关键一步。