This study investigates the forecasting accuracy of human experts versus Large Language Models (LLMs) in the retail sector, particularly during standard and promotional sales periods. Utilizing a controlled experimental setup with 123 human forecasters and five LLMs, including ChatGPT4, ChatGPT3.5, Bard, Bing, and Llama2, we evaluated forecasting precision through Mean Absolute Percentage Error. Our analysis centered on the effect of the following factors on forecasters performance: the supporting statistical model (baseline and advanced), whether the product was on promotion, and the nature of external impact. The findings indicate that LLMs do not consistently outperform humans in forecasting accuracy and that advanced statistical forecasting models do not uniformly enhance the performance of either human forecasters or LLMs. Both human and LLM forecasters exhibited increased forecasting errors, particularly during promotional periods and under the influence of positive external impacts. Our findings call for careful consideration when integrating LLMs into practical forecasting processes.

翻译：本研究探究了零售领域中，人类专家与大型语言模型（LLMs）在常规销售期和促销销售期的预测准确性。通过一个包含123名人类预测者和五种LLM（包括ChatGPT4、ChatGPT3.5、Bard、Bing和Llama2）的受控实验设置，我们利用平均绝对百分比误差评估了预测精度。我们的分析聚焦于以下因素对预测者表现的影响：所采用的统计模型（基础模型与高级模型）、产品是否处于促销状态，以及外部冲击的性质。研究结果表明，在预测准确性方面，LLMs并不总是优于人类；同时，高级统计预测模型也并未均一提升人类预测者或LLMs的表现。人类和LLM预测者在促销期间以及正向外部冲击影响下，均表现出更高的预测误差。我们的发现呼吁，在将LLMs整合到实际预测流程中时需审慎考量。