The growing prominence of large language models, such as GPT-4 and ChatGPT, has led to increased concerns over academic integrity due to the potential for machine-generated content and paraphrasing. Although studies have explored the detection of human- and machine-paraphrased content, the comparison between these types of content remains underexplored. In this paper, we conduct a comprehensive analysis of various datasets commonly employed for paraphrase detection tasks and evaluate an array of detection methods. Our findings highlight the strengths and limitations of different detection methods in terms of performance on individual datasets, revealing a lack of suitable machine-generated datasets that can be aligned with human expectations. Our main finding is that human-authored paraphrases exceed machine-generated ones in terms of difficulty, diversity, and similarity implying that automatically generated texts are not yet on par with human-level performance. Transformers emerged as the most effective method across datasets with TF-IDF excelling on semantically diverse corpora. Additionally, we identify four datasets as the most diverse and challenging for paraphrase detection.

翻译：大型语言模型（如GPT-4和ChatGPT）的日益普及引发了学术界对学术诚信的担忧，因为机器生成内容和同义改写存在潜在风险。尽管已有研究探讨了人类改写内容与机器改写内容的检测，但这两类内容之间的对比仍未被充分探索。本文对同义改写检测任务中常用的多个数据集进行了全面分析，并评估了一系列检测方法。我们的研究结果揭示了不同检测方法在单个数据集上的性能优势与局限性，同时指出目前缺乏能够符合人类预期的合适机器生成数据集。主要发现是，人类撰写的同义改写内容在难度、多样性和相似性方面均优于机器生成的内容，这表明自动生成的文本尚未达到人类水平的表现。Transformer在所有数据集中被证明是最有效的方法，而TF-IDF在语义多样性语料库上表现突出。此外，我们识别出四个数据集是同义改写检测中最多样化且最具挑战性的。