Small Variance, Big Fairness: A Path to Harmless Fairness without Demographics

TL;DR — Harmless Fairness without Demographics via decreasing Variance of Losses

Abstract

Statistical fairness harnesses a classifier to accommodate parity requirements by equalizing the model utility (e.g., accuracy) across disadvantaged and advantaged groups. Due to privacy and security concerns, recently there has arisen a need for learning fair classifiers without ready-to-use demographic information. Existing studies remedy this challenge by introducing various side information about groups and many of them are found fair by unavoidably comprising model utility. $Can\ we\ improve\ fairness\ without\ demographics\ and\ without\ hurting\ model\ utility?$ To address this problem, we propose to center on minimizing the variance of losses, allowing the model to effectively eliminate possible accuracy disparities without knowledge of sensitive attributes. During optimization, we develop a dynamic harmless update approach operating at both loss and gradient levels, directing the model towards fair solutions while preserving its intact utility. Through extensive experiments across four benchmark datasets, our results consistently demonstrate that our method effectively reduces group accuracy disparities while maintaining comparable or even improved utility.