The impact of random seeds in fine-tuning large language models (LLMs) has been largely overlooked despite its potential influence on model performance.In this study, we systematically evaluate the effects of random seeds on LLMs using the GLUE and SuperGLUE benchmarks. We analyze the macro-level impact through traditional metrics like accuracy and F1, calculating their mean and variance to quantify performance fluctuations. To capture the micro-level effects, we introduce a novel metric, consistency, measuring the stability of individual predictions across runs. Our experiments reveal significant variance at both macro and micro levels, underscoring the need for careful consideration of random seeds in fine-tuning and evaluation.

翻译：尽管随机种子可能对模型性能产生潜在影响，但在微调大语言模型（LLMs）过程中其作用长期被忽视。本研究基于GLUE和SuperGLUE基准，系统评估了随机种子对LLMs的影响。我们通过准确率和F1值等传统指标分析宏观层面的影响，计算其均值与方差以量化性能波动。为捕捉微观层面的效应，我们引入了一种新颖的指标——一致性，用于衡量多次运行中个体预测的稳定性。实验结果表明，在宏观与微观层面均存在显著方差，这凸显了在微调与评估过程中需审慎考量随机种子的重要性。