Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.

翻译：大规模语言模型（LLMs）已在软件工程（SE）领域获得显著应用，影响从代码补全到测试生成、从程序修复到代码摘要等多项SE任务。尽管其前景广阔，研究人员仍需保持谨慎，因为诸多复杂因素可能影响涉及LLMs的实验结果。本文旨在开启关于LLM研究有效性潜在威胁的公开讨论，包括闭源模型、LLM训练数据与研究评估之间可能存在的数泄漏，以及基于LLM研究结果的可复现性问题。为此，本文提出一套专为SE研究人员和语言模型（LM）供应商量身定制的指导方针，以缓解这些担忧。通过结合LLM供应商遵循的现有良好实践以及一个在测试用例生成背景下SE研究人员的实际示例，本文阐述了这些指导方针的意义。