Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.

翻译：大型语言模型（LLMs）已在软件工程社区中获得显著关注，影响着从代码补全到测试生成、从程序修复到代码摘要等各类软件工程任务。尽管前景广阔，研究人员仍需谨慎，因为众多复杂因素可能影响涉及LLMs的实验结果。本文发起了一场公开讨论，探讨基于LLM的研究面临的潜在有效性威胁，包括闭源模型、LLM训练数据与研究评估之间可能的数据泄漏，以及基于LLM的研究结果的可复现性。为此，本文提出了一套针对软件工程研究人员和语言模型（LM）提供者的指导方针，以缓解这些问题。这些指导方针的意义通过LLM提供者现有的良好实践以及面向软件工程研究人员在测试用例生成场景下的实际示例加以说明。