How Neural Networks With Derivative Labels Work: A Neural Tangent Kernel Perspective

Abstract

Deep neural networks have achieved impressive results in a range of fields, while their analytical properties have been slow to develop. Recently, a theoretical tool called Neural Tangent Kernel (NTK) has been proposed and extended to diverse architectures based on neural networks. This tool helps explain their convergence and generalization with a least-square loss. However, researchers in numerous fields have trained their networks using an additional derivative loss item, such as Jacobian Regularization and PINN (Physics-Informed Neural Networks). This loss setup has generality, while it often leads to challenging convergence issues. To address this problem, we propose a general paradigm that utilizes Gâteaux derivative labels to describe all these tasks. Additionally, we extend NTK in our setup from an analytical perspective and propose a geometrical perspective of parameter updating directions to explain the hard convergence. We also conduct experiments to verify our propositions. Finally, we provide specific expressions for distinct tasks within our paradigm.