MetriQ: A Fair Data Pruning Method for Robust Deep Learning

Table of Links

Abstract and 1 Introduction

2 Data Pruning & Fairness

3 Method & Results

4 Theoretical Analysis

5 Discussion, Acknowledgments and Disclosure of Funding, and References

A Implementation Details

B Theoretical Analysis for a Mixture of Gaussians

3 Method & Results

We are ready to propose our “fairness-aware” data pruning method, which consists in random subsampling according to carefully selected target class-wise sizes. To imitate the behavior of the cost-sensitive learning procedures that optimize for worst-class accuracy as discussed in Section 2, we propose to select the pruning fraction of each class based on its validation performance given a preliminary model ψ trained on the whole dataset. This query model is still required by all existing data pruning algorithms to compute scores, so we introduce no additional resource overhead.

Evaluation. To validate the effectiveness of random pruning with MetriQ (Random+MetriQ) in reducing classification bias, we compare it to various baselines derived from two of the strongest pruning algorithms: GraNd [Paul et al., 2021], and Forgetting [Toneva et al., 2019]. In addition to plain random pruning (Random)—removing a random subset of all training samples—for each of these two strategies, we consider (1) random pruning that respects class-wise ratios automatically determined by the strategy (Random+StrategyQ), (2) applying the strategy for pruning within classes but distributing sample quotas across classes according to MetriQ (Strategy+MetriQ), and (3) the strategy itself. The implementation details and hyperparameter choices are documented in Appendix A.

Results. Figure 4 presents our results for (a) VGG16 on CIFAR-10, (b) VGG-19 on CIFAR-100, and (c) ResNet-18 on TinyImageNet. Overall, MetriQ with random pruning consistently exhibits a significant improvement in distributional robustness of the trained models. In contrast to all other baselines that arguably achieve their highest worst-class accuracy at high dataset densities (80–90%), our method reduces classification bias induced by the datasets as pruning continues, e.g., up to 30–40% dataset density of TinyImageNet. Notably, Random+MetriQ improves all fairnes metrics compared to the full dataset on all model-dataset pairs, offering both

Figure 4: The average test performance of various data pruning protocols against dataset density and measures of class-wise fairness. All results averaged over 3 random seeds. Error bands represent min/max.

robustness and data efficiency at the same time. For example, when pruning half of CIFAR-100, we achieve an increase in the worst-class accuracy of VGG-19 from 35.8% to 45.4%—an almost 10% change at a price of under 6% of the average performance. The leftmost plots in Figure 4 reveal that Random+MetriQ does suffer a slightly larger degradation of the average accuracy as dataset density decreases compared to global random pruning, which is unavoidable given the natural trade-off between robustness and average performance. Yet at these low densities, the average accuracy of our method exceeds that of all other pruning algorithms.

As demonstrated in Figure 5, MetriQ produces exceptionally imbalanced datasets unless the density d is too low by heavily pruning easy classes while leaving the more difficult ones intact. As expected from its design, the negative correlation between the full-dataset accuracy and density of each class is much more pronounced for MetriQ compared to existing pruning methods (cf. Figure 3). Based on our examination of these methods in Section 2, we conjectured that these two properties are associated with smaller classification bias, which is well-supported by MetriQ. Not only does it achieve unparalleled performance with random pruning, but it also enhances fairness of GraNd and Forgetting: Strategy+MetriQ curves trace a much better trade-off between the average and worst-class accuracies than their original counterparts (Strategy). At the same time, Random+StrategyQ fares similarly well, surpassing the vanilla algorithms, too. This indicates that robustness is achieved not only from the appropriate class ratios but also from pruning randomly as opposed to cherry-picking difficult samples. We develop more analytical intuition about this phenomenon in Section 4.

Having access to the full dataset, CDB-W improves the worst-class accuracy of VGG-19 on CIFAR100 by 5.7% compared to standard optimization, which is almost twice as short of the increase obtained by removing 50% of the dataset with Random+MetriQ (Figure 4). When applied to EL2Npruned datasets, CDB-W retains that original fairness level across sparsities, which is clearly inferior not only to Random+MetriQ but also to EL2N+MetriQ. Perhaps surprisingly, EL2N with scores computed by a query model trained with CDB-W fails spectacularly, inducing one of the worst bias observed in this study. Thus, MetriQ can compete with other existing methods that directly optimize for worst-class accuracy.