Learning Forward Compatible Representation in Class Incremental Learning by Increasing Effective Rank

Abstract

Class Incremental Learning (CIL) is a prominent subfield of continual learning, aiming to enable models to incrementally learn new tasks while preserving the knowledge learned from the previous tasks. The main challenge of CIL is known as catastrophic forgetting, where a model that is naively fine-tuned to new tasks experiences a significant drop in performance on previous tasks. To address the challenge, previous studies have mostly focused on backward compatible approaches. Recently, a forward compatible approach has been introduced that supports a concurrent use with the existing backward compatible methods. The forward compatible method, however, is limited in that it relies solely on class information. In this study, we propose an effective-Rank based Forward Compatible (RFC) representation regularization that is not confined to specific types of information, such as class information. The proposed method increases the efficient rank of representation during the base session, thereby facilitating the encoding of more informative features pertinent to unseen novel tasks. To substantiate the effectiveness of our method, we establish a theoretical connection between effective rank and Shannon entropy of the representations. Subsequently, we conduct comprehensive experiments, by integrating it into ten well-known backward compatible CIL methods. The results demonstrate that our forward compatible approach is effective in enhancing the performance of novel tasks while mitigating catastrophic forgetting. Furthermore, the results indicate that our method significantly improves the average incremental accuracy of all ten cases that we have examined, underscoring its efficacy and general applicability.