Text continues to remain a relevant form of representation for information. Text documents are created either in digital native platforms or through the conversion of other media files such as images and speech. While the digital native text is invariably obtained through physical or virtual keyboards, technologies such as OCR and speech recognition are utilized to transform the images and speech signals into text content. All these variety of mechanisms of text generation also introduce errors into the captured text. This project aims at analyzing different kinds of error that occurs in text documents. The work employs two of the advanced deep neural network-based language models, namely, BART and MarianMT, to rectify the anomalies present in the text. Transfer learning of these models with available dataset is performed to finetune their capacity for error correction. A comparative study is conducted to investigate the effectiveness of these models in handling each of the defined error categories. It is observed that while both models can bring down the erroneous sentences by 20+%, BART can handle spelling errors far better (24.6%) than grammatical errors (8.8%).
翻译:文本仍然是信息表示的一种重要形式。文本文档或通过数字原生平台创建,或通过其他媒体文件(如图像和语音)的转换生成。虽然数字原生文本通常通过物理或虚拟键盘获取,但光学字符识别和语音识别等技术被用于将图像和语音信号转换为文本内容。所有这些文本生成机制也都会在捕获的文本中引入错误。本项目旨在分析文本文档中出现的不同类型的错误。研究采用了两种先进的深度神经网络语言模型,即BART和MarianMT,来纠正文本中存在的异常。通过使用现有数据集对这些模型进行迁移学习,以微调其错误纠正能力。我们进行了一项比较研究,以探究这些模型在处理每种定义错误类别时的有效性。结果表明,尽管两个模型都能将错误句子减少20%以上,但BART处理拼写错误(24.6%)的能力远优于处理语法错误(8.8%)。