LLM (large language model) practitioners commonly notice that outputs can vary for the same inputs under settings expected to be deterministic. Yet the questions of how pervasive this is, and with what impact on results, have not to our knowledge been systematically investigated. We investigate non-determinism in five LLMs configured to be deterministic when applied to eight common tasks in across 10 runs, in both zero-shot and few-shot settings. We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%. In fact, none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings. Sharing preliminary results with insiders has revealed that non-determinism perhaps essential to the efficient use of compute resources via co-mingled data in input buffers so this issue is not going away anytime soon. To better quantify our observations, we introduce metrics focused on quantifying determinism, TARr@N for the total agreement rate at N runs over raw output, and TARa@N for total agreement rate of parsed-out answers. Our code and data are publicly available at http://github.com/REDACTED.

翻译：大语言模型（LLM）实践者常注意到，在预期应为确定性的设置下，相同输入可能产生不同的输出。然而，据我们所知，这种现象的普遍程度及其对结果的影响尚未得到系统研究。我们在零样本和少样本设置下，对五种配置为确定性的LLM在八个常见任务上进行了10次运行，以研究其非确定性。我们观察到自然运行中准确率波动高达15%，最佳与最差性能差距可达70%。事实上，所有LLM均无法在所有任务中保持可重复的准确率，更不用说完全一致的输出字符串。与业内人士分享的初步结果表明，非确定性可能源于输入缓冲区中数据混合以实现计算资源高效利用的必要机制，因此该问题短期内不会消失。为量化观测结果，我们提出了专注于衡量确定性的指标：TARr@N（原始输出在N次运行中的总一致率）和TARa@N（解析答案的总一致率）。我们的代码与数据公开于http://github.com/REDACTED。