Gradient norm as a powerful proxy to out-of-distribution error estimation

TL;DR — We use the norm of classification-layer gradients, backpropagated from the cross entropy loss with only one gradient step over OOD data, to formulate an estimation score correlating with the expected OOD error.

Abstract

Estimating out-of-distribution (OOD) error without access to the ground-truth test labels is a highly challenging, yet extremely important problem in the safe deployment of machine learning algorithms. Current works rely on the information from either the outputs or the extracted features to formulate an estimation score correlating with the expected OOD error. In this paper, we investigate--both empirically and theoretically--how the information provided by the gradients can be predictive of the OOD error. Specifically, we use the norm of classification-layer gradients, backpropagated from the cross-entropy loss with only one gradient step over OOD data. Our key idea is that the model should be adjusted with a higher magnitude of gradients when it does not generalize to the OOD dataset. We provide theoretical insights highlighting the main ingredients of such an approach ensuring its empirical success. Extensive experiments conducted on diverse distribution shifts and model structures demonstrate that our method outperforms state-of-the-art algorithms significantly.