Neural scaling laws, which in some domains can predict the performance of large neural networks as a function of model, data, and compute scale, are the cornerstone of building foundation models in Natural Language Processing and Computer Vision. We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting. To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture, and we use continual training with constant learning rates and periodic cooldowns as an efficient training strategy. We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules. Cooldown phases can be re-purposed to improve downstream performance, e.g., enabling accurate multi-step rollouts over longer forecast horizons as well as sharper predictions through spectral loss adjustments. We also systematically explore a wide range of model and dataset sizes under various compute budgets to construct IsoFLOP curves, and we identify compute-optimal training regimes. Extrapolating these trends to larger scales highlights potential performance limits, demonstrating that neural scaling can serve as an important diagnostic for efficient resource allocation. We open-source our code for reproducibility.

翻译：神经缩放定律作为自然语言处理和计算机视觉领域构建基础模型的基石，能在某些场景中根据模型规模、数据量和计算量预测大型神经网络的性能。本研究聚焦天气预报模型，探讨科学机器学习中的神经缩放现象。为在尽可能简化的设定下分析缩放行为，我们采用最小化、可扩展的通用Swin Transformer架构，并运用恒定学习率与周期性冷却阶段的持续训练策略作为高效训练方案。研究表明，采用这种极简方式训练的模型可遵循可预测的缩放趋势，其表现甚至优于标准余弦学习率调度方法。冷却阶段可被重新利用以提升下游任务性能，例如实现更长期预报窗口下的精准多步推演，以及通过频谱损失调整获得更锐利的预测结果。我们还在不同计算预算下系统探索了广泛的模型与数据集规模组合，构建了IsoFLOP曲线并识别出计算最优训练方案。将这些趋势外推至更大规模后显现了潜在性能边界，证明神经缩放可作为资源高效配置的重要诊断工具。为保障可复现性，我们开源了全部代码。