Pick-or-Mix: Dynamic Channel Sampling for ConvNets

TL;DR — A single dynamic channel sampling module targeting various tasks at once i.e. channel squeezing, network downscaling, dynamic pruning

Abstract

Channel squeezing is a crucial operation in convolutional neural networks (ConvNets). It is carried out via 1 × 1 convolution layers and dominates a large portion of computations and parameters of a given network. ResNet-50, for instance, consists of 16 such layers, forming 33% of total layers and 25% (1.05B/4.12B) of total FLOPs. In light of their predominance, we present a new multi-purpose module for dynamic channel sampling, namely Pick-or-Mix (PiX). PiX divides a set of channels into subsets and then picks from them, where the picking decision is dynamically made per each pixel based on the input activations. We show that PiX allows ConvNets to learn better data representation than vanilla channel squeezing in far fewer computations. We plug PiX into prominent ConvNet architectures and verify its multi-purpose utilities. After replacing 1 × 1 channel squeezing layers in the ResNet family with PiX, the networks become 25% faster without losing accuracy. We also show that PiX can achieve state-of-the-art performance on network downscaling and dynamic channel pruning.