Dynamical stability and chaos in artificial neural network trajectories along training

• DOI: 10.3389/fcpxs.2024.1367957

The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network’s prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space–a time series of networks–and thus the training algorithm (e.g., gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.

• Full paper

Related research projects

INFOLANET

Information processing with coupled laser networks

P.I.: Apostolos Argyris, Miguel C. Soriano

In the INFOLANET project, we will combine the expertise of the PIs on dynamical systems and machine learning to advance information processing concepts, based on a high-speed photonic implementation. We anticipate that …

MdM-IFISC-2

Maria de Maeztu 2023-2026

P.I.: Ernesto Estrada, Ingo Fischer, Emilio Hernández-García, Rosa Lopez, Claudio Mirasso, Jose Javier Ramasco, Raúl Toral, Roberta Zambrini

After 15 years of its existence, IFISC can point to a proven track record of impactful research. The previous 2018-2022 MdM award has significantly enhanced the institute's capabilities, as demonstrated by an …

DYNDEEP

Dynamics of Temporal Networks: Memory and Deep Learning

P.I.: Lucas Lacasa

The interaction between elements of a complex system arising in physics, biology or sociology can be modelled as a mathematical graph. The precise architecture of this interaction backbone plays a fundamental role …