Accelerated Inference and Reduced Forgetting: The Dual Benefits of Early-Exit Networks in Continual Learning

Abstract

In the pursuit of a sustainable future for machine learning, energy-efficient neural network models are crucial. A practical approach to achieving this efficiency is through early-exit strategies. These strategies allow for swift predictions by making decisions early in the network, thereby conserving computation time and resources. However, so far the early-exit neural networks have only been developed for stationary data distributions, which restricts their application in real-world scenarios where training data is derived from continuous non-stationary data. In this study, we aim to explore the continual training for early-exit networks. Specifically, we adapt the existing continual learning methods to fit early-exit architectures and introduce task-aware dynamic inference to improve the network accuracy for a given compute budgets. Finally, we evaluate continually those methods on the standard benchmarks to assess their accuracy and efficiency. Our work highlights the practical advantages of the early-exit networks in real-world continual learning scenarios.