Scalar Representations of Neural Network Training Dynamics

• DOI: https://doi.org/10.48550/arXiv.2606.30384
• ArXiv number: 2606.30384

Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging. In this work, we treat such training trajectories as temporal networks and apply recently proposed strategies for the scalar embedding of temporal networks. We investigate whether such a scalar embedding provides a meaningful low-dimensional representation of neural network training dynamics. Using a multilayer perceptron trained on the MNIST classification task, we show that the embedding preserves the main dynamical features observed in the original parameter space, including the emergence of sensitivity to initial conditions for specific learning rate regimes and an accurate reconstruction of the network's maximum Lyapunov exponent. We then use the embedded scalar trajectory to define a characteristic time, analogous to a Lyapunov time, after which the exponential separation between initially close embedded trajectories saturates. This characteristic time captures the typical decorrelation time between initially close network trajectories in the original high-dimensional system. Finally, we investigate the statistical organization of asymptotic training states through a spacing observable defined in the embedded space. We find that the distributions of rescaled asymptotic spacings collapse onto a common form across initial conditions and are compatible with a skew lognormal distribution. Altogether, our results suggest that scalar low-dimensional embeddings provide a useful framework for studying and visualizing the dynamical properties of neural network optimization trajectories.

• Artículo completo

Related research projects

CSxAI

Complexity science for understanding AI: from the dynamics of complex networks to collective effects of interacting neural networks

I.P.: Lucas Lacasa, Víctor M. Eguíluz

The CSxAI project engages researchers at IFISC in a (non-oriented research project) on the interdisciplinary challenge of applying tools and concepts of complexity science (dynamical systems, statistical physics, and network science) to …

NEHIL

Neuromorphic-Enhanced Heterogeneously-Integrated FMCW LiDAR

I.P.: Miguel C. Soriano

The NEHIL project, an EU-Korea partnership, is set to transform the landscape of digital technologies through groundbreaking neuromorphic architectures and advanced heterogeneous integration such as LiDAR systems. This collaborative initiative aims to …

MdM-IFISC-2

Maria de Maeztu 2023-2026

I.P.: Ernesto Estrada, Ingo Fischer, Emilio Hernández-García, Rosa Lopez, Claudio Mirasso, Jose Javier Ramasco, Raúl Toral, Roberta Zambrini

After 15 years of its existence, IFISC can point to a proven track record of impactful research. The previous 2018-2022 MdM award has significantly enhanced the institute's capabilities, as demonstrated by an …