Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.

翻译：优化在现代机器学习（ML）与科学机器学习（SciML）中均处于核心地位，然而这两个领域中底层优化问题的结构存在显著差异。经典机器学习通常依赖于随机、样本可分离的目标函数，这有利于一阶及自适应梯度方法。相比之下，科学机器学习常涉及物理信息或算子约束的表述，其中微分算子会导致损失函数景观中的全局耦合、刚性及强各向异性。因此，科学机器学习中的优化行为主要由底层物理模型的光谱特性而非数据统计特性所主导，这常常限制了标准随机方法的有效性，并促使人们采用确定性或曲率感知的优化方法。本文档对机器学习和科学机器学习中的优化方法进行了统一介绍，重点阐述了问题结构如何影响算法选择。我们回顾了确定性与随机设置下的一阶和二阶优化技术，讨论了它们在物理约束和数据驱动的科学机器学习模型中的适应性，并通过教程示例说明了实用策略，同时强调了科学计算与科学机器学习交叉领域中的开放研究方向。