Linking Finite-Time Lyapunov Exponents to RNN Gradient Subspaces and Input Sensitivity

Abstract

Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. They can be viewed as non-autonomous dynamical systems which can be analyzed using dynamical systems tools, such as Lyapunov Exponents. In this work, we derive and analyze the components of RNNs' Finite Time Lyapunov Exponents (FTLE) which measure directions (vectors Q) and factors (scalars R) with which the distance between nearby trajectories expands or contracts over finite-time horizons. We derive an expression for RNN gradients in terms of $Q$ vectors and $R$ values and demonstrate a direct connection between these quantities and the loss. We find that the dominant directions of the gradient extracted by singular value decomposition become increasingly aligned with the dominant $Q$ vectors as training proceeds. Furthermore, we show that the task outcome of an RNN is maximally affected by input perturbations at moments where high state space expansion is taking place (as measured by FTLEs). Our results showcase deep links between computations, loss gradients, and dynamical systems stability theory for RNNs. This opens the way to design adaptive methods that take into account state-space dynamics to improve computations.