Leveraging chaotic transients in the training of artificial neural networks

• DOI: https://doi.org/10.1103/t5p9-kv5w
• ArXiv number: 2506.08523

Traditional algorithms to optimize artificial neural networks when confronted with a supervised learning task are usually exploitation-type relaxational dynamics such as gradient descent (GD). Here, we explore the dynamics of the neural network trajectory along training for unconventionally large learning rates. We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitation-like algorithm into a regime of exploration-exploitation balance, as the neural network is still capable of learning but the trajectory shows sensitive dependence on initial conditions –as characterized by positive network maximum Lyapunov exponent–. Interestingly, the characteristic training time required to reach an acceptable accuracy in the test set reaches a minimum precisely in such learning rate region, further suggesting that one can accelerate the training of artificial neural networks by locating at the onset of chaos. Our results
–initially illustrated for the MNIST classification task– qualitatively hold for a range of supervised learning tasks, learning architectures (including both shallow and deep multilayer perceptrons and convolutional neural networks) and other hyperparameters (different activation functions and weight regularisation), and showcase the emergent, constructive role of transient chaotic dynamics in the
training of artificial neural networks.

• Full paper

Related research projects

CSxAI

Complexity science for understanding AI: from the dynamics of complex networks to collective effects of interacting neural networks

P.I.: Lucas Lacasa, Víctor M. Eguíluz

The CSxAI project engages researchers at IFISC in a (non-oriented research project) on the interdisciplinary challenge of applying tools and concepts of complexity science (dynamical systems, statistical physics, and network science) to …

NEHIL

Neuromorphic-Enhanced Heterogeneously-Integrated FMCW LiDAR

P.I.: Miguel C. Soriano

The NEHIL project, an EU-Korea partnership, is set to transform the landscape of digital technologies through groundbreaking neuromorphic architectures and advanced heterogeneous integration such as LiDAR systems. This collaborative initiative aims to …

MdM-IFISC-2

Maria de Maeztu 2023-2026

P.I.: Ernesto Estrada, Ingo Fischer, Emilio Hernández-García, Rosa Lopez, Claudio Mirasso, Jose Javier Ramasco, Raúl Toral, Roberta Zambrini

After 15 years of its existence, IFISC can point to a proven track record of impactful research. The previous 2018-2022 MdM award has significantly enhanced the institute's capabilities, as demonstrated by an …

Related News

A touch of chaos can help artificial intelligence learn faster

May 25, 2026

The study, published in Physical Review Research, shows that neural networks can train faster when their learning dynamics operate at the onset of chaotic behavior, balancing exploration and exploitation during optimization.The study was carried out by researchers at the Institute …

Un toc de caos pot ajudar la intel·ligència artificial a aprendre més aviat

May 25, 2026

L'estudi, publicat a Physical Review Research, mostra que les xarxes neuronals es poden entrenar més aviat quan la seva dinàmica d'aprenentatge opera en l'inici del comportament caòtic, i s’equilibren l'exploració i l'explotació durant l'optimització.L'estudi ha estat realitzat per investigadors de …

Un toque de caos puede ayudar a la inteligencia artificial a aprender más rápido

May 25, 2026

El estudio, publicado en Physical Review Research, muestra que las redes neuronales pueden entrenarse más rápido cuando su dinámica de aprendizaje opera en el inicio del comportamiento caótico, equilibrando la exploración y la explotación durante la optimización.El estudio fue realizado …