Compressed representations in deep learning: From algorithmic information to autoencoders and hypernetworks

In this thesis, the deep connection between intelligence and compression is explored, emphasizing how compression principles give rise to the principles of deep learning. First an overview of ideal compression is provided, based on the theory of computation, with particular focus on Solomonoff theory of induction. From there the minimum description length principle can be derived. This leads to two part codes, where in virtue of the concentration of measure, lead to the minimization of a bound on Shannon entropy thus leading to lossless compression. Two more contributions are present in this work. First, compressed representations in the context of lossy compression, are investigated in latent variable generative models using variational inference. A particular focus is taken in the analysis of the quantized representations of the vector quantized variational autoencoders (VQ-VAEs), where it is shown that there is a connection to Hopfield networks through a continuous relaxation of the quantization procedure. This connection can be used to provide an improvement in the training dynamics when compared to previous methods, by learning richer representations. The other contribution consists of using the concepts from compressed representations and combine them with a variant of meta learning, to design a novel architecture that predicts future frames of the moving MNIST dataset. This last result shows that instead of directly learning a function to model those transitions, hypernetworks can be used to leverage compressed representations to hypothesize how future frames will unfold. In a nutshell, this work explores the theoretical underpinnings of compression and uses them to provide novel generative algorithms.