Sparling: Learning Latent Representations With Extremely Sparse Activations

TL;DR — We introduce Sparling, an extreme activation sparsity layer and optimization algorithm that, when used in the middle of a neural network trained purely end-to-end, can recover the ground-truth intermediate features.

Abstract

Real-world processes often contain intermediate state that can be modeled as an extremely sparse tensor. We introduce Sparling, a technique that allows you to learn models with intermediate layers that match this state from only end-to-end labeled examples (i.e., no supervision on the intermediate state). Sparling uses a new kind of informational bottleneck that enforces levels of activation sparsity unachievable using other techniques. We find that extreme sparsity is necessary to achieve good intermediate state modeling. On our synthetic DigitCircle domain as well as the LaTeXOCR and AudioMNISTSequence domains, we are able to precisely localize the intermediate states up to feature permutation with $>90\%$ accuracy, even though we only train end-to-end.