Rethinking Self-Supervise Learning: An Instance-wise Similarity Perspective

TL;DR — We propose a novel self-supervised learning approach, to learn an appropriately sparse IwS matrix in the representation space

Abstract

This paper studies self-supervised learning from the perspective of instance-wise similarity (IwS), characterized by the pairwise similarity matrix among all instances. Ideally, the IwS matrix in the representation space should closely mirror that in the input space so that the learned representations retain their discriminative power and account for semantic similarities. This perspective not only allows us to understand diverse existing self-supervised learning methodologies better but also uncovers a notable limitation within current approaches: the discrepancy between IwS matrices in the input and representation spaces. Indeed, many established methods, including SimCLR and MoCo v3, implicitly assume that the IwS matrix within the representation space is an identity matrix, even when the IwS matrix in the input space may deviate from this form. Inspired by this observation, we introduce sparse contrastive learning, a new approach that learns an appropriately sparse IwS matrix within the representation space instead of presuming an identity IwS matrix. Our comprehensive experiments conducted on ImageNet and CIFAR datasets substantiate the superior performance of our method in comparison to other state-of-the-art methods.