Statistical Inference for Deep Learning via Stochastic Modeling

TL;DR — We develop an innovative framework for performing statistical inference for deep learning models.

Abstract

Deep learning has revolutionized big data analysis in modern data science, however, how to make statistical inference for deep neural networks remains largely unclear. To this end, we explore a stochastic variant of the deep neural network known as the stochastic neural network (StoNet). Firstly, we show that the StoNet falls into the framework of statistical modeling. It not only enables us to address fundamental issues in deep learning, such as structure interpretability and uncertainty quantification, but also provides with us a platform for transferring the theory and methods developed for linear models to the realm of deep learning. Specifically, we show how the sparse learning theory with the Lasso penalty can be adapted to deep neural networks (DNNs) from linear models; establish that the sparse StoNet is consistent in network structure selection; and provides a recursive method to quantify the prediction uncertainty for the Stonet. Furthermore, we extend this result to the DNN by its asymptotic equivalence with the StoNet, showing that consistent sparse deep learning can be obtained by training a DNN with an appropriate Lasso penalty. Additionally, we propose to remodel the last hidden layer output and the target output of a well-trained DNN model using a StoNet on the validation dataset, and then assess the prediction uncertainty of the DNN model via the Stonet. The proposed method has been compared with conformal inference on extensive examples, and numerical results suggests its superiority.