MoteS: Memory Optimization via Fine-grained Scheduling for DNNs on Tiny Devices

TL;DR — We propose a memory optimizer for DNN depolyment on tiny devices via fine-grained graph scheduling

Abstract

There has been a growing trend in deploying deep neural networks (DNNs) on tiny devices. However, it is challenging to do so due to the contradiction of large execution memory requirement of many DNNs and stringent memory constraint of tiny devices. Some previous works incurs large latency overhead to save memory and cannot optimize networks with complex structures; some employ coarse-grained scheduling, leading to limited memory footprint reduction. This paper proposes MoteS that performs fine-grained scheduling via operator partitioning on DNNs to dramatically reduce peak memory usage with little latency overhead. MoteS presents a graph representation named Axis Connecting Graph (ACG) to perform operator partition at graph-level efficiently. MoteS further proposes an algorithm that searches the partition and schedule guided by memory bottlenecks. We evaluate MoteS using various popular networks and show that MoteS achieves up to 80\% of peak memory usage reduction compared to state-of-art works with nearly no latency overhead on tiny devices.