Title: A Primer on Optimization Strategies for Training Scientific Machine Learning Models

Abstract:

While optimization serves as a cornerstone for both contemporary machine learning (ML) and scientific machine learning (SciML), the fundamental nature of the optimization challenges in these fields diverges significantly. Traditional ML predominantly utilizes stochastic objectives that are separable by sample, a structure that makes first-order and adaptive gradient techniques particularly effective. Conversely, SciML frequently employs formulations constrained by physics or operators, where differential operators create global coupling, stiffness, and pronounced anisotropy within the loss landscape. Consequently, the optimization dynamics in SciML are dictated by the spectral characteristics of the physical models at hand, rather than by statistical properties of the data. This distinction often diminishes the utility of conventional stochastic methods, thereby necessitating the use of deterministic or curvature-aware algorithms. This paper offers a cohesive overview of optimization methodologies in both ML and SciML, focusing on how the specific structure of a problem influences the selection of algorithms. We examine first- and second-order techniques applicable to both deterministic and stochastic contexts, explore their adaptation for physics-constrained and data-driven SciML frameworks, and demonstrate practical implementation strategies via tutorial examples. Additionally, the discussion outlines emerging research opportunities at the intersection of scientific computing and scientific machine learning.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC