Improving Robustness and Accuracy with Retrospective Online Adversarial Distillation

TL;DR — We propose retrospective online adversarial distillation (ROAD), to improve robustness against adversarial attacks and natural accuracy.

Abstract

Adversarial distillation (AD), transferring knowledge of a robust teacher model to a student model, has emerged as an advanced technique for improving robustness against adversarial attacks. However, AD in general suffers from the high computational complexity of pre-training the robust teacher, and the inherent trade-off between robustness and natural accuracy (i.e., accuracy on clean data). To address these issues, we propose retrospective online adversarial distillation (ROAD). ROAD exploits the student itself of the last epoch and a natural model (i.e., a model trained with clean data) as teachers, instead of a pre-trained robust teacher in the conventional AD. We revealed both theoretically and empirically that knowledge distillation from the student of the last epoch allows to penalize overly confident predictions on adversarial examples, leading to improved robustness and generalization. Also, the student and the natural model are trained together in a collaborative manner, which enables to improve natural accuracy of the student more effectively. We demonstrate by extensive experiments that ROAD achieved outstanding performance in both robustness and natural accuracy with substantially reduced training time and computation cost.