The learning-chaos trade-off in neural network training

Using tools from dynamical systems and network science, we investigate the internal mechanisms governing the training dynamics of multilayer perceptrons (MLPs). In particular, we examine the behaviour of the trajectory of network parameters under unusually high learning rates. Our analysis reveals a regime in which the optimisation trajectory shows a strong sensitivity to initial conditions, characterised by a transition to positive maximum Lyapunov exponents and the emergence of chaotic dynamics. Surprisingly, we observe that the training time reaches a minimum within this chaotic regime, suggesting that this onset of chaos can enhance learning efficiency. These results are validated on the MNIST classification task and shown to be generalisable to other supervised learning tasks, architectures and hyperparameter configurations.