Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors

Abstract

Conventional Federated Learning (FL) involves collaborative training of a global model by multiple client local models. In this emerging paradigm, the central server assumes a critical role in aggregating local models and maintaining the global model. However, it encounters various challenges, including scalability, management, and inefficiencies arising from idle client devices. Recently, studies on serverless decentralized FL have shown advantages in overcoming these challenges, enabling clients to own different local models and separately optimize local data. Despite the promising advancements in decentralized FL, it is crucial to thoroughly investigate the implications of data and model heterogeneity, which pose unique challenges that must be overcome. Therefore, the research question to be answered in this study is: How can every client's local model learn generalizable representation? To address this question, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Inspired by the theory of domain adaptation and Knowledge distillation (KD), we leverage the synthetic anchors to design two effective regularization terms for local training: 1) anchor loss that matches the distribution of the client's latent embedding with an anchor and 2) KD loss that enables clients learning from others. In contrast to previous KD-based heterogeneous FL methods, we don’t presume access to real public or a global data generator. DeSA enables each client's model to become robust to distribution shift across different client-domains. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of \ours{} in enhancing both inter and intra-domain accuracy of each client.