Continual Nonlinear ICA-Based Representation Learning

TL;DR — We develop a novel approach for nonlinear ICA that effectively accommodates continually arriving domains and show its identifiability from subspace to component-wise level.

Abstract

Unsupervised identification of disentangled representations remains a challenging problem. Recent progress in nonlinear Independent Component Analysis (ICA) provides a promising causal representation learning framework by separating latent sources from observable nonlinear mixtures. However, its identifiability hinges on the incorporation of side information, such as time or domain indexes, which are challenging to obtain adequately offline in real-world scenarios. In this paper, we develop a novel approach for nonlinear ICA that effectively accommodates continually arriving domains. We first theoretically demonstrate that model identifiability escalates from subspace to component-wise identifiability as new domains are involved. It motivates us to maintain prior knowledge and progressively refine it using new arriving domains. Upon observing a new domain, our approach optimizes the model by satisfying two objectives: (1) reconstructing the observations within the current domain, and (2) preserving the reconstruction capabilities for prior domains through gradient constraints. Experiments demonstrate that our method achieves performance comparable to nonlinear ICA methods trained jointly on multiple offline domains, demonstrating its practical applicability in continual learning scenarios.