Test like you Train in Implicit Deep Learning

TL;DR — We show that Deep Equilibrium Models (DEQs) do not in practice benefit from a higher number of inner iterations at test-time compared to that used in training.

Abstract

Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks~(DEQs). In its very general formulation, it relies on expressing some components of deep learning pipelines implicitly, typically via a root equation called the inner problem. In practice, the solution of the inner problem is approximated with an iterative procedure, usually with a fixed number of inner iterations during training. At inference time, the inner problems needs to be solved with new data. A popular belief is that increasing the number of inner iterations relative to the one used in training yields better performances. In this paper, we question such an assumption and provide a detailed theoretical analysis in a simple affine setting. We demonstrate that overparametrization plays a key role: increasing the number of iterations at test time cannot improve performances for overparametrized networks. We validate our theory on an array of implicit deep-learning problems. We show that DEQs, which are typically overparametrized, do not benefit from increasing the number of iterations at inference while meta-learning, which is typically not overparametrized, benefits from it.