Document similarity is an important part of Natural Language Processing and is most commonly used for plagiarism-detection and text summarization. Thus, finding the overall most effective document similarity algorithm could have a major positive impact on the field of Natural Language Processing. This report sets out to examine the numerous document similarity algorithms, and determine which ones are the most useful. It addresses the most effective document similarity algorithm by categorizing them into 3 types of document similarity algorithms: statistical algorithms, neural networks, and corpus/knowledge-based algorithms. The most effective algorithms in each category are also compared in our work using a series of benchmark datasets and evaluations that test every possible area that each algorithm could be used in.
翻译:文档相似度是自然语言处理的重要组成部分,最常用于抄袭检测和文本摘要。因此,寻找整体最优的文档相似度算法可能对自然语言处理领域产生重大积极影响。本文旨在系统研究多种文档相似度算法,并确定其中最有用的方法。通过将文档相似度算法分为三类(统计算法、神经网络算法、基于语料库/知识的算法),本文探讨了最有效的文档相似度算法。此外,我们利用一系列基准数据集和评估方法对每类中最优算法进行了比较,这些评估覆盖了各算法可能应用的各个领域。