ProtoReg: Prioritizing Discriminative Information for Fine-grained Transfer Learning

TL;DR — We propose a simple yet effective method that utilizes adaptively evolving class prototypes to capture fine-grained information.

Abstract

Transfer learning leverages a pre-trained model with rich features to fine-tune it for downstream tasks, thereby improving generalization performance. However, we point out the "granularity gap" in fine-grained transfer learning, a mismatch between the level of information learned by a pre-trained model and the semantic details required for a fine-grained downstream task. Under these circumstances, excessive non-discriminative information can hinder the sufficient learning of discriminative semantic details. In this study, we address this issue by establishing class-discriminative prototypes and refining the prototypes to gradually encapsulate more fine-grained semantic details, while explicitly aggregating each feature with the corresponding prototype. This approach allows the model to prioritize fine-grained discriminative information, even when the pre-trained model contains excessive non-discriminative information due to the granularity gap. Our proposed simple yet effective method, ProtoReg, significantly outperforms other transfer learning methods in fine-grained classification benchmarks with an average performance improvement of 6.4\% compared to standard fine-tuning. Particularly in limited data scenarios using only 15\% of the training data, ProtoReg achieves an even more substantial average improvement of 13.4\%. Furthermore, ProtoReg demonstrates robustness to shortcut learning when evaluated on out-of-distribution data.