Fluency is a crucial goal of all Natural Language Generation (NLG) systems. Widely used automatic evaluation metrics fall short in capturing the fluency of machine-generated text. Assessing the fluency of NLG systems poses a challenge since these models are not limited to simply reusing words from the input but may also generate abstractions. Existing reference-based fluency evaluations, such as word overlap measures, often exhibit weak correlations with human judgments. This paper adapts an existing unsupervised technique for measuring text fluency without the need for any reference. Our approach leverages various word embeddings and trains language models using Recurrent Neural Network (RNN) architectures. We also experiment with other available multilingual Language Models (LMs). To assess the performance of the models, we conduct a comparative analysis across 10 Indic languages, correlating the obtained fluency scores with human judgments. Our code and human-annotated benchmark test-set for fluency is available at https://github.com/AnanyaCoder/TextFluencyForIndicLanaguges.

翻译：流畅度是所有自然语言生成（NLG）系统的核心目标。广泛使用的自动评估指标在捕捉机器生成文本的流畅度方面存在明显不足。评估NLG系统的流畅度具有挑战性，因为这类模型不仅限于重复输入词汇，还可能生成抽象表达。现有的基于参考标准的流畅度评估方法（如词汇重叠测量）往往与人工判断的相关性较弱。本文改编了一种无需任何参考标准的现有无监督技术来衡量文本流畅度。该方法利用多种词嵌入，并通过循环神经网络（RNN）架构训练语言模型。我们还对其他可用的多语言语言模型（LMs）进行了实验。为评估模型性能，我们在10种印度语言上展开对比分析，并将获得的流畅度评分与人工判断进行相关性验证。我们的代码及人工标注的流畅度基准测试集可访问：https://github.com/AnanyaCoder/TextFluencyForIndicLanaguges。