Accurate load forecasting plays a vital role in numerous sectors, but accurately capturing the complex dynamics of dynamic power systems remains a challenge for traditional statistical models. For these reasons, time-series models (ARIMA) and deep-learning models (ANN, LSTM, GRU, etc.) are commonly deployed and often experience higher success. In this paper, we analyze the efficacy of the recently developed Transformer-based Neural Network model in Load forecasting. Transformer models have the potential to improve Load forecasting because of their ability to learn long-range dependencies derived from their Attention Mechanism. We apply several metaheuristics namely Differential Evolution to find the optimal hyperparameters of the Transformer-based Neural Network to produce accurate forecasts. Differential Evolution provides scalable, robust, global solutions to non-differentiable, multi-objective, or constrained optimization problems. Our work compares the proposed Transformer based Neural Network model integrated with different metaheuristic algorithms by their performance in Load forecasting based on numerical metrics such as Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE). Our findings demonstrate the potential of metaheuristic-enhanced Transformer-based Neural Network models in Load forecasting accuracy and provide optimal hyperparameters for each model.

翻译：准确的负荷预测在众多领域中都扮演着关键角色，但传统统计模型难以准确捕捉动态电力系统的复杂动态特性。因此，时间序列模型（ARIMA）和深度学习模型（ANN、LSTM、GRU等）被广泛采用并通常取得较高成功率。本文分析了最新开发的基于Transformer的神经网络模型在负荷预测中的有效性。Transformer模型由于其注意力机制能够学习长距离依赖关系，因此有望改善负荷预测性能。我们应用多种元启发式算法（特别是差分进化算法）来寻找基于Transformer的神经网络模型的最优超参数，以实现准确预测。差分进化算法能够为非可微、多目标或约束优化问题提供可扩展、鲁棒且全局的解决方案。本研究通过均方误差（MSE）和平均绝对百分比误差（MAPE）等数值指标，比较了所提出的基于Transformer的神经网络模型与不同元启发式算法集成后的负荷预测性能。我们的研究结果展示了元启发式增强的基于Transformer的神经网络模型在提高负荷预测精度方面的潜力，并为每个模型提供了最优超参数。