AdaSR: Adaptive Super Resolution for Cross Platform and Dynamic Runtime Environments

TL;DR — We present the framework AdaSR which allows for super-resolution models to adapt inference latencies at run time in order to accommodate dynamic resource availability and cross-platform deployments.

Abstract

Image super resolution models (SR) have shown great capability in improving the visual quality for low-resolution images. Due to the compute and memory budgets of diverse platforms, e.g., cloud and edge devices, practitioners and researchers have to either (1) design different architectures and/or (2) compress the same model to different levels. Additionally, a majority of the works in current literature aim to achieve state-of-the-art performance by hand-designing singular efficient models. However, even on the same hardware, the compute resource dynamics change due to other running applications. As such, one single model that satisfies required frames-per-second (FPS) when executed in isolation may not be suitable when other running applications present. To overcome those issues, we propose AdaSR, an Adaptive SR framework via shared architecture and weights for cross platform deployment and dynamic runtime environment. Unlike other works in literature, our work focuses on the development of multiple models within a larger meta-graph such that they can fulfill latency requirements by compromising as little performance as possible. Particularly, AdaSR can be used to (1) customize architectures for different hardware (e.g., different security cameras), and (2) adaptively change the compute graph in dynamic runtime environment (e.g., mobile phones with concurrently running applications). Different than prior arts, AdaSR achieves this by adaptively changing the depth and the channel size with shared weights and architecture, which introduces no extra cost on memory and/or storage. To stabilize the shared weight training of AdaSR, we propose a progressive approach where we derive loss functions for each block and function matching operations with max-norm regularization to address dimension mismatches. We extensively test AdaSR on different block-based GAN models, and demonstrate that AdaSR can maintain Pareto optimal performance in terms of latency vs. performance tradeoff with much smaller memory footprint and support dynamic runtime environments.