Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods

Overview

• The paper explores techniques for "pruning" or compressing modern large language models (LLMs) to reduce their high computational needs.
• It compares two main approaches: "width pruning" (reducing the size of projection weight matrices) and "depth pruning" (removing entire layers or blocks).
• The key finding is that simple depth pruning can effectively compress LLMs while achieving comparable or superior performance to recent width pruning methods.

Plain English Explanation

Large language models (LLMs) like GPT-3 and BERT are incredibly powerful, but they also require a lot of computing power to run. This makes them difficult to deploy in resource-constrained environments like mobile devices. To address this, researchers have been exploring ways to "prune" or compress these models without losing too much performance.

The paper looks at two main approaches to pruning:

1. Width pruning: This involves reducing the size of the weight matrices used in the model, for example by removing some of the attention heads. This can shrink the model's overall size, but the number of layers remains the same.

2. Depth pruning: This method removes entire layers or blocks from the model, while keeping the remaining weights unchanged. This can achieve more dramatic model compression.

The key finding is that simple depth pruning can be just as effective as more complex width pruning techniques, and in some cases even better. Depth pruning was especially helpful for improving inference speed (how quickly the model can make predictions) in situations where memory is limited, and the model has to run with small batch sizes.

When retraining the pruned models to recover performance, the researchers found that continued pretraining on a large corpus was much more effective than a technique called LoRA, particularly for models that were heavily pruned.

Overall, this work suggests that depth pruning could be a simpler and more effective way to build compact yet capable language models, especially for deployment on resource-constrained devices.

Technical Explanation

The paper explores two main approaches to structured pruning of large language models (LLMs):

1. Width pruning: This involves reducing the size of the projection weight matrices, such as by removing attention heads. The number of layers in the model remains unchanged.

2. Depth pruning: This method removes entire layers or blocks from the model, while keeping the size of the remaining weights unchanged. This can achieve more dramatic model compression.

Most prior research has focused on width-only pruning or a blend of width and depth pruning, with little comparative analysis between the two.

In this work, the authors show that simple depth pruning can effectively compress LLMs while achieving comparable or superior performance to recent width pruning studies. Their depth pruning method boosts inference speeds, especially under memory-constrained conditions that require limited batch sizes, where width pruning is less effective.

When retraining the pruned models to recover performance, the authors found that continued pretraining on a large corpus markedly outperforms LoRA-based tuning, particularly at severe pruning ratios.

Critical Analysis

The paper provides a thorough evaluation of depth pruning as an alternative to more commonly studied width pruning techniques. The authors acknowledge that their depth pruning method is a relatively simple approach, but argue that it can be just as effective as more complex methods.

One potential limitation of the research is that it primarily focuses on comparing depth and width pruning, without exploring more advanced pruning strategies that combine the two techniques, as mentioned in related work. It would be interesting to see how a hybrid approach might perform.

Additionally, the paper does not delve into the theoretical reasons why depth pruning may be more effective than width pruning in certain scenarios, such as memory-constrained inference. Further analysis of the underlying mechanisms could provide valuable insights.

Overall, the research makes a compelling case for depth pruning as a simple yet powerful technique for compressing large language models. The findings could have important implications for deploying these models in resource-constrained environments, and the authors' code and models provide a useful starting point for further exploration.

Conclusion

This paper demonstrates that simple depth pruning can be an effective and efficient way to compress large language models, often outperforming more complex width pruning techniques. The depth pruning method boosts inference speeds, especially in memory-constrained conditions, and continued pretraining is shown to be more effective than LoRA-based tuning for recovering model performance after pruning.

These findings suggest that depth pruning could be a promising approach for building compact yet capable language models, particularly for deployment on resource-constrained devices. The work contributes to the ongoing efforts to make large language models more accessible and practical for a wider range of applications.