Compact Model Training by Low-Rank Projection with Energy Transfer

Overview

• Low-rank compression is an important technique in traditional machine learning, but has not been as popular in deep learning.
• Previous low-rank compression methods compressed pre-trained models, but the optimal solution with a low-rank constraint may be different from the original.
• Retraining a low-rank compressed model often leads to significant performance degradation.
• Low-rank compression has attracted less attention compared to other methods like pruning in recent years.

Plain English Explanation

Low-rank compression is a way to make machine learning models smaller and more efficient by approximating the original model's weights with a lower-dimensional representation. This technique has been commonly used in traditional machine learning, but hasn't been as widely adopted in the world of deep learning.

The reason for this is that previous low-rank compression methods would start with a well-trained deep learning model, then try to compress it by finding a low-rank approximation of the model's weight matrices. However, this optimal low-rank solution may be quite different from the original unconstrained weights. As a result, retraining the compressed model often leads to a significant drop in performance.

In contrast, other model compression techniques like pruning have become more popular in deep learning in recent years. These methods can often preserve model performance better than the traditional low-rank approaches.

Technical Explanation

In this paper, the authors propose a new training method called "Low-Rank Projection with Energy Transfer" (LRPET) that can train low-rank compressed deep learning models from scratch, without the need for pretraining.

The key ideas are:

1. Alternating optimization: They alternate between performing stochastic gradient descent training and projecting each weight matrix onto a low-rank manifold.
2. Energy transfer: The reduction in "energy" (sum of squared singular values) caused by the low-rank projection is compensated by uniformly transferring the "energy" of the pruned singular values to the remaining ones.
3. BN rectification: In modern deep networks, batch normalization (BN) layers can influence the optimal low-rank approximation of the preceding convolution layers. The authors propose "BN rectification" to address this issue.

By training the low-rank model from scratch and using energy transfer, the authors are able to better utilize the model capacity and avoid the significant performance degradation seen in previous low-rank compression methods.

Critical Analysis

The authors provide a thorough theoretical analysis to support their claims about the benefits of their LRPET training method. However, some potential limitations or areas for further research include:

• Evaluating the LRPET method on a wider range of deep learning architectures and tasks beyond the image classification experiments presented.
• Investigating the computational and memory efficiency of the LRPET training process compared to other compression techniques like pruning or low-rank decomposition.
• Exploring the interaction between low-rank compression and other model optimization techniques, such as model quantization.

Overall, the LRPET method represents an interesting approach to training low-rank compressed deep learning models, and the authors' theoretical analysis provides a solid foundation for further research in this area.

Conclusion

This paper presents a new training method called LRPET that can effectively train low-rank compressed deep learning models from scratch, avoiding the performance degradation seen in previous approaches. By alternating optimization with low-rank projections and using an energy transfer mechanism, LRPET is able to better utilize the model capacity within the low-rank constraint. The authors also introduce BN rectification to address the influence of batch normalization layers on the optimal low-rank approximation.

While low-rank compression has been less popular in deep learning compared to other techniques like pruning, the LRPET method demonstrates the potential for low-rank approaches to achieve competitive performance. Further research exploring the practical benefits and limitations of LRPET across a wider range of deep learning applications could help drive more widespread adoption of low-rank compression in the field.