Understanding Deep Neural Networks as Dynamical Systems: Insights into Training and Fine-tuning

Abstract

This paper offers an interpretation mechanism for understanding deep neural networks and their learning processes from a dynamical perspective. The aim is to uncover the relationship between the representational capacity of neural networks and the dynamical properties of their corresponding dynamical systems. To this end, we first interpret neural networks as dynamical systems by representing neural weight values as relationships among neuronal dynamics. Then, we model both neural network training and inference as the dynamical phenomena occurring within these systems. Built upon this framework, we introduce the concept of dynamical discrepancy, a macroscopic attribute that describes the dynamical states of neurons. Taking the generalization capability of neural models as a starting point, we launch a hypothesis: the dynamical discrepancy of neuromorphic-dynamical systems correlates with the representational capacity of neural models. We conduct dynamics-based conversions on neural structures such as ResNet, ViT, and LLaMA to investigate this hypothesis on MNIST, ImageNet, SQuAD, and IMDB. The experimental fact reveals that the relationship between these neural models' dynamical discrepancy and representational capacity aligns perfectly with our theoretical conjecture. Building upon these findings, we introduce a universal analytical approach tailored for neural models.