Spelling error correction is the task of identifying and rectifying misspelled words in texts. It is a potential and active research topic in Natural Language Processing because of numerous applications in human language understanding. The phonetically or visually similar yet semantically distinct characters make it an arduous task in any language. Earlier efforts on spelling error correction in Bangla and resource-scarce Indic languages focused on rule-based, statistical, and machine learning-based methods which we found rather inefficient. In particular, machine learning-based approaches, which exhibit superior performance to rule-based and statistical methods, are ineffective as they correct each character regardless of its appropriateness. In this paper, we propose a novel detector-purificator-corrector framework, DPCSpell based on denoising transformers by addressing previous issues. In addition to that, we present a method for large-scale corpus creation from scratch which in turn resolves the resource limitation problem of any left-to-right scripted language. The empirical outcomes demonstrate the effectiveness of our approach, which outperforms previous state-of-the-art methods by attaining an exact match (EM) score of 94.78%, a precision score of 0.9487, a recall score of 0.9478, an f1 score of 0.948, an f0.5 score of 0.9483, and a modified accuracy (MA) score of 95.16% for Bangla spelling error correction. The models and corpus are publicly available at https://tinyurl.com/DPCSpell.
翻译:拼写错误纠正是识别并纠正文本中错误拼写单词的任务。由于其在人类语言理解中的广泛应用,它已成为自然语言处理中一个具有潜力且活跃的研究课题。语音或视觉上相似但语义不同的字符使得该任务在任何语言中都极具挑战性。早期针对孟加拉语及资源稀缺印度语言的拼写纠错工作主要集中于基于规则、统计和机器学习的方法,我们发现这些方法效率较低。特别是,虽然基于机器学习的方法在性能上优于基于规则和统计的方法,但它们因不加区分地纠正每个字符而效果不佳。在本文中,我们提出了一种新颖的检测器-净化器-校正器框架DPCSpell,该框架基于去噪Transformer,旨在解决先前存在的问题。此外,我们还提出了一种从零开始创建大规模语料库的方法,从而解决了任何从左到右书写语言的资源限制问题。实证结果表明了我们方法的有效性,在孟加拉语拼写纠错任务中,我们的方法以94.78%的精确匹配(EM)分数、0.9487的精确率、0.9478的召回率、0.948的f1分数、0.9483的f0.5分数以及95.16%的修正准确率(MA),超越了先前最先进的方法。模型和语料库已在 https://tinyurl.com/DPCSpell 公开提供。