Nowadays both commercial and open-source academic LLM have become the mainstream models of NLP. However, there is still a lack of research on LLM consistency, meaning that throughout the various stages of LLM research and deployment, its internal parameters and capabilities should remain unchanged. This issue exists in both the industrial and academic sectors. The solution to this problem is often time-consuming and labor-intensive, and there is also an additional cost of secondary deployment, resulting in economic and time losses. To fill this gap, we build an LLM consistency task dataset and design several baselines. Additionally, we choose models of diverse scales for the main experiments. Specifically, in the LightGBM experiment, we used traditional NLG metrics (i.e., ROUGE, BLEU, METEOR) as the features needed for model training. The final result exceeds the manual evaluation and GPT3.5 as well as other models in the main experiment, achieving the best performance. In the end, we use the best performing LightGBM model as the base model to build the evaluation tool, which can effectively assist in the deployment of business models. Our code and tool demo are available at https://github.com/heavenhellchen/Consistency.git

翻译：如今，无论是商业还是开源学术型大语言模型（LLM）已成为自然语言处理（NLP）领域的主流模型。然而，针对LLM一致性的研究仍存在不足，即在其研发与部署的各个阶段，其内部参数与能力应保持稳定。这一问题在工业界与学术界均有体现。解决该问题通常耗时耗力，且存在二次部署的额外成本，从而造成经济与时间损失。为弥补这一空白，我们构建了LLM一致性任务数据集，并设计了多个基线模型。此外，我们选取了不同规模的模型进行主实验。具体而言，在LightGBM实验中，我们采用传统自然语言生成指标（如ROUGE、BLEU、METEOR）作为模型训练所需的特征。最终结果超越了人工评估、GPT3.5及主实验中其他模型，取得了最佳性能。最后，我们将性能最优的LightGBM模型作为基础模型构建评估工具，该工具可有效辅助商业模型部署。我们的代码与工具演示见https://github.com/heavenhellchen/Consistency.git