Heterogeneity of Regularization between adjacent periods

TL;DR — This paper proposes Periodic Regularization and integrates this method with Reinforcement Learning or Transfer Learning to establish regularization techniques that consistently demonstrates remarkable performance across multiple datasets

Abstract

Since the inception of deep learning, regularization techniques have been developed for the purpose of preventing the overfitting phenomenon. Regularization is typically accomplished in two ways: incorporating randomness (e.g., injecting noise into data, activating nodes, or using dropout) or heterogeneity (e.g., data augmentation). These approaches are known to lead to better generalization and, consequently, improved performance. In the case of introducing heterogeneity by adjusting the hyperparameter during the training process, such as the drop rate of dropout, experiments have shown that tuning hyperparameters after a period, which consists of a certain number of forward propagations, is more effective than either uniformly sustaining hyperparameters or tuning them during every propagation. Therefore, this paper proposes a novel regularization technique named Periodic Regularization that introduces periodicity into the dynamic hyperparameter tuning of other regularization methods. Furthermore, this paper suggests combining Periodic Regularization and other learning techniques such as Reinforcement Learning and Transfer Learning. This approach, particularly when combining dropout and reinforcement learning, shows significant improvement in empirical testing across various popular datasets. This is notably evident in Facial Expression Recognition (FER) tasks, where conventional methods, such as noise injection and dropout, have proven ineffective. Our proposed periodic regularization method not only can fill the research gap found in traditional regularization techniques but also can be a cornerstone for further research where the concept of periodic regularization is combined with diverse vanilla regularization techniques and learning techniques.