Task-Distributionally Robust Data-Free Meta-Learning

Abstract

Data-free Meta-learning (DFML) aims to enable efficient learning of new unseen tasks by meta-learning from a collection of pre-trained models without access to their training data. Existing DFML methods construct pseudo tasks from a learnable dataset, which is iteratively inversed from a collection of pre-trained models. However, such distribution of pseudo tasks is not stationary and can be easily corrupted by specific attack, which causes (i) Task-Distribution Shift (TDS): the distribution of tasks will change as the learnable dataset gets updated, making the meta-learner biased and susceptible to overfitting on new tasks, ultimately harming its long-term generalization performance. (ii) Task-Distribution Corruption (TDC): the task distribution can be easily corrupted by deliberately injecting deceptive out-of-distribution models, termed as model poisoning attack. To address these issues, for the first time, we call for and develop robust DFML. Specifically, (i) for handling TDS, we propose a new memory-based DFML baseline (TEAPOT) via meta-learning from a pseudo task distribution. TEAPOT maintains the memory of old tasks to prevent over-reliance on new tasks, with an interpolation mechanism combining classes from different tasks to diversify the pseudo task distribution; (ii) for further defending against TDC, we propose a defense strategy, Robust Model Selection Policy (ROSY), which is compatible with existing DFML methods (e.g., ROSY + TEAPOT). ROSY adaptively ranks and then selects reliable models according to a learnable reliability score, which is optimized by policy gradient due to the non-differentiable property of model selection. Extensive experiments show the superiority of TEAPOT over existing baselines for handling TDS and verify the effectiveness of ROSY + DFML for further improving robustness against TDC.