Neural Architecture Search via Two Constant Shared Weights Initialisations

Overview

• In recent years, zero-cost metrics have gained traction in neural architecture search (NAS)
• These metrics allow finding optimal neural networks faster and with less computational load than traditional NAS methods
• They also provide insights into the internal workings of neural architectures
• This paper presents a zero-cost metric that is highly correlated with train set accuracy across multiple NAS benchmarks

Plain English Explanation

Designing the optimal neural network architecture for a given task can be a time-consuming and computationally-intensive process. Zero-cost metrics offer a potential solution by allowing researchers to quickly evaluate the potential of a neural architecture without fully training it.

The key idea behind this paper is to assess a neural network's performance based on the statistics of the outputs after two constant weight initializations. The researchers found that the dispersion of the outputs between these two initializations positively correlates with the eventual trained accuracy of the network. Normalizing this dispersion by the average output magnitude further improves the correlation.

This resulting metric, called epsilon, does not require gradient computations or human-labeled data. It can be quickly evaluated and easily integrated into existing NAS algorithms. By bypassing the need for training, epsilon helps accelerate the NAS process and provides insights into the inner workings of neural architectures without the overhead of full training.

Technical Explanation

The key contribution of this paper is a zero-cost metric, called epsilon, that is highly correlated with the train set accuracy of neural architectures across multiple NAS benchmark datasets (NAS-Bench-101, NAS-Bench-201, and NAS-Bench-NLP).

To compute epsilon, the researchers evaluate a neural network's potential based on the outputs' statistics after two constant shared weight initializations. Specifically, they use an unlabeled mini-batch of data to observe the dispersion of the network's outputs between the two initializations. They find that this dispersion positively correlates with the eventual trained accuracy of the network.

Further improvements are made by normalizing the dispersion by the average output magnitude. This normalized dispersion metric does not require gradient computations or human-labeled data, allowing it to be quickly evaluated.

The researchers demonstrate that epsilon can be easily integrated into existing NAS algorithms and takes only a fraction of a second to evaluate a single network. This makes it a promising tool for accelerating the NAS process and providing insights into the internal representations of neural architectures without the computational overhead of full training.

Critical Analysis

The researchers acknowledge several limitations of their approach. First, the correlation between epsilon and trained accuracy, while strong, is not perfect. This means that epsilon cannot fully replace traditional NAS methods but rather serves as a useful complement to quickly identify promising architectures.

Additionally, the researchers only evaluate epsilon on image classification and language modeling tasks. Its performance on other domains, such as reinforcement learning or generative modeling, remains to be seen.

Another potential concern is the reliance on constant weight initializations. While this allows for a fast evaluation, it may not fully capture the impact of more complex weight initialization schemes used in practice.

Despite these limitations, the epsilon metric represents a valuable contribution to the field of NAS. By providing a computationally-efficient way to evaluate neural architectures, it has the potential to accelerate research and development in areas that rely on neural networks.

Conclusion

This paper introduces a zero-cost metric, called epsilon, that is highly correlated with the train set accuracy of neural architectures. By evaluating the dispersion of a network's outputs after two constant weight initializations, epsilon provides a fast and computationally-efficient way to assess the potential of a neural architecture without the need for full training or human-labeled data.

While epsilon has limitations and cannot fully replace traditional NAS methods, it represents a promising step towards more efficient and insightful neural architecture search. By providing insights into the internal workings of neural networks and accelerating the NAS process, epsilon has the potential to drive significant advances in the field of deep learning.