Self-supervised Dataset Distillation: A Good Compression Is All You Need

Overview

• Dataset distillation compresses information from a large dataset into a smaller dataset while preserving the original data's essence.
• Previous studies focused on aligning intermediate model statistics like weight trajectory, features, and batch normalization between the original and distilled data.
• This work explores a new perspective: model informativeness during the compression stage on original dataset pretraining.

Plain English Explanation

The goal of dataset distillation is to take a large, complex dataset and create a smaller, more compact version that still captures the important information from the original. This can be useful for training machine learning models more efficiently, as the smaller dataset requires less computational power and storage.

Previous approaches to dataset distillation have mainly tried to ensure that certain statistical properties of the original dataset, like the trajectory of the model's weights or the features it learns, are preserved in the distilled dataset. However, the authors of this paper noticed that as model sizes increase, it becomes harder for supervised pre-trained models to recover the full scope of information during the data synthesis (or distillation) process. This is because the mean and variance of the model's internal channels tend to flatten out, making the model less informative.

The key insight here is that larger variances in the batch normalization statistics of self-supervised models allow for stronger learning signals to update the recovered data during synthesis. Building on this, the researchers introduce a new framework called SC-DD, which uses self-supervision to facilitate more diverse and effective information compression and recovery compared to traditional supervised approaches.

Technical Explanation

The researchers observe that as model sizes increase, supervised dataset distillation methods like SRe²L, MTT, TESLA, DC, and CAFE struggle to fully recover the information learned by the original large pre-trained models. This is because the channel-wise mean and variance inside the models tend to flatten out, making the models less informative.

In contrast, the authors notice that self-supervised models have larger variances in their batch normalization statistics, which enables stronger learning signals to update the recovered data during the synthesis process. Building on this, they introduce SC-DD, a self-supervised compression framework for dataset distillation. SC-DD leverages the enhanced informative capabilities of large self-supervised pre-trained models to achieve more effective data compression and recovery compared to traditional supervised approaches.

The researchers conduct extensive experiments on CIFAR-100, Tiny-ImageNet, and ImageNet-1K datasets, demonstrating that SC-DD outperforms previous state-of-the-art supervised dataset distillation methods when using larger models, across various recovery and post-training budgets.

Critical Analysis

The paper provides a novel perspective on dataset distillation by focusing on model informativeness during the compression stage. The authors make a compelling case for the benefits of leveraging self-supervised pre-training to enhance the distillation process, especially as model sizes increase.

However, the paper does not discuss potential limitations or caveats of the SC-DD approach. For example, it would be valuable to understand how the method performs on more diverse or challenging datasets, or how it scales to even larger model sizes. Additionally, the authors could explore the trade-offs between the computational and memory requirements of self-supervised pre-training versus the gains in distillation performance.

Further research could also investigate the specific mechanisms by which self-supervised models maintain greater informative capacity during the distillation process. Understanding these underlying dynamics could lead to additional improvements in dataset distillation techniques.

Conclusion

This paper introduces a new self-supervised framework for dataset distillation, called SC-DD, that leverages the enhanced informative capabilities of large self-supervised pre-trained models to achieve more effective data compression and recovery compared to traditional supervised approaches. The researchers demonstrate the superiority of SC-DD over previous state-of-the-art methods, particularly when using larger models. This work highlights the importance of considering model informativeness during the dataset distillation process and opens up new avenues for further research in this area.