Prompt engineering relevance research has seen a notable surge in recent years, primarily driven by advancements in pre-trained language models and large language models. However, a critical issue has been identified within this domain: the inadequate of sensitivity and robustness of these models towards Prompt Templates, particularly in lesser-studied languages such as Japanese. This paper explores this issue through a comprehensive evaluation of several representative Large Language Models (LLMs) and a widely-utilized pre-trained model(PLM). These models are scrutinized using a benchmark dataset in Japanese, with the aim to assess and analyze the performance of the current multilingual models in this context. Our experimental results reveal startling discrepancies. A simple modification in the sentence structure of the Prompt Template led to a drastic drop in the accuracy of GPT-4 from 49.21 to 25.44. This observation underscores the fact that even the highly performance GPT-4 model encounters significant stability issues when dealing with diverse Japanese prompt templates, rendering the consistency of the model's output results questionable. In light of these findings, we conclude by proposing potential research trajectories to further enhance the development and performance of Large Language Models in their current stage.

翻译：近年来，随着预训练语言模型和大语言模型的快速发展，提示工程相关研究显著增加。然而，该领域已发现一个关键问题：这些模型对提示模板的敏感性和鲁棒性不足，尤其是在日语等研究较少的语言中。本文通过系统评估多个代表性大语言模型（LLMs）及广泛使用的预训练模型（PLM）来探究这一问题。我们使用日语基准数据集对这些模型进行检验，旨在评估并分析当前多语言模型在此场景中的表现。实验结果显示令人震惊的差异：仅对提示模板的句子结构进行简单修改，就导致GPT-4的准确率从49.21%骤降至25.44%。这一发现表明，即使是性能卓越的GPT-4模型，在处理多样化的日语提示模板时仍面临显著稳定性问题，使得模型输出结果的一致性存疑。基于上述发现，我们最后提出潜在的研究方向，以进一步推动大语言模型在当前阶段的发展与性能提升。