Effective theory of collective deep learning

• DOI: https://doi.org/10.1103/PhysRevResearch.6.L042040
• ArXiv number: 2310.12802

Unraveling the emergence of collective learning in systems of coupled artificial neural networks points to broader implications for physics, machine learning, neuroscience, and society. Here we introduce a minimal model of interacting deep neural nets that condenses several recent decentralized algorithms by considering a competition between two terms: the local learning dynamics in the parameters of each neural network unit, and a diffusive coupling among units that tends to homogenize the parameters of the ensemble. We derive an effective theory for linear networks to show that the coarse-grained behavior of our system is equivalent to a deformed Ginzburg-Landau model with quenched disorder. This framework predicts depth-dependent disorder-order-disorder phase transitions in the parameters’ solutions that reveal a depth-delayed onset of a collective learning phase and a low-rank microscopic learning path. We validate the theory in coupled ensembles of realistic neural networks trained on MNIST and CIFAR-10 datasets under privacy constraints. Interestingly, experiments confirm that individual networks–trained on private data–can fully generalize to unseen data classes when the collective learning phase emerges. Our work elucidates the physics of collective learning and contributes to the mechanistic interpretability of deep learning in decentralized settings.

• Article complet

Related research projects

NEHIL

Neuromorphic-Enhanced Heterogeneously-Integrated FMCW LiDAR

I.P.: Miguel C. Soriano

The NEHIL project, an EU-Korea partnership, is set to transform the landscape of digital technologies through groundbreaking neuromorphic architectures and advanced heterogeneous integration such as LiDAR systems. This collaborative initiative aims to …

DYNDEEP

Dynamics of Temporal Networks: Memory and Deep Learning

I.P.: Lucas Lacasa

The interaction between elements of a complex system arising in physics, biology or sociology can be modelled as a mathematical graph. The precise architecture of this interaction backbone plays a fundamental role …

MISLAND

Modelling island ecological complexity in the context of global change

I.P.: Lucas Lacasa, Víctor M. Eguíluz

** This project (PID2020-114324GB-C22) is part of a coordinated project between IFISC and IMEDEA, both research centers from CSIC located in Mallorca. The project is funded by AEI and a PhD fellowship …