LayerNorm: A key component in parameter-efficient fine-tuning

Introduction

The paper proposes a parameter-efficient fine-tuning approach for transformer-based language models like BERT. Despite their excellent performance, these models are computationally expensive to fine-tune due to their large number of parameters. Existing methods like only training bias parameters or a small portion of the model aim to reduce this cost.

The authors hypothesize that focusing on the optimal component of the model can achieve similar or better performance with fewer parameters. They examine different components of BERT during full fine-tuning and discover that LayerNorm is a key component. LayerNorm possesses the maximum Fisher information among all BERT components.

The authors demonstrate that training only LayerNorm can reach similar performance as training only bias parameters, but with one-fifth the number of parameters. They further show that comparable performance can be obtained by training only a portion of LayerNorm, which can be determined from the downstream task or other tasks.

The paper is organized as follows: Section 2 shows LayerNorm is key for fine-tuning BERT. Section 3 presents the method and results of training only LayerNorm. Section 4 discusses the findings, while Section 5 reviews related work. Section 6 provides conclusions and future work. An appendix describes LayerNorm in detail.

A key component of BERT

The paper aims to identify the most important components of the BERT model for fine-tuning. The authors used the GLUE dataset and fine-tuned BERT-large-cased on different tasks. After fine-tuning, they calculated the change in each component's parameters from their pre-trained values. This change was measured using the L1 distance normalized by the component size.

The heat maps showed that the LayerNorm component underwent the most significant change across most GLUE tasks. Previous studies have also shown that disabling LayerNorm severely degrades BERT's performance.

The authors then used Fisher information to quantify each component's importance. They calculated the Fisher information of each parameter, averaged it for each component, and normalized across tasks. The LayerNorm and attention.output.LayerNorm components had the highest Fisher information, confirming their importance.

In summary, the LayerNorm component appears to be the most crucial for fine-tuning BERT, based on the analysis of parameter changes and Fisher information across GLUE tasks.

Proposed method: Only training LayerNorm

The paper explores fine-tuning only the LayerNorm parameters of a pre-trained BERT model for various GLUE tasks. Experiments show that fine-tuning just the LayerNorm parameters achieves comparable performance to fine-tuning all parameters, but with only a fraction (0.015%) of trainable parameters.

Further analysis is done by training only a subset of the LayerNorm parameters, selected based on their Fisher information score. Results indicate that even training 20% of the LayerNorm parameters can maintain good performance on some tasks.

Visualizations reveal that higher layers of LayerNorm contain more task-relevant information than lower layers, and the bias terms have higher Fisher scores than weights.

The paper also explores finding a global subset of LayerNorm parameters that performs well across all tasks, instead of a task-specific subset. This global subset achieves similar performance to the task-specific subsets through cross-validation experiments.

Overall, the key finding is that fine-tuning a small subset of LayerNorm parameters in pre-trained models can be an extremely parameter-efficient way to adapt to new tasks while retaining performance.

Discussion

The section discusses the importance of the LayerNorm component in Transformer-based models like BERT. Disabling LayerNorm significantly degrades the performance of these models. During fine-tuning, LayerNorm undergoes greater changes compared to other components. Training only LayerNorm or a small portion of it achieves comparable performance to fine-tuning the full model, despite its smaller parameter size.

The analysis shows that the bias terms in LayerNorm contain more information than the weight terms across various GLUE tasks. This aligns with previous findings that training only the bias terms of BERT can be effective.

The study observes that the final layers of BERT's LayerNorm have higher Fisher information (indicating larger gradients) compared to the initial layers. This trend exists across GLUE tasks, suggesting that the final layers undergo more changes during fine-tuning. This phenomenon has been observed in other studies as well.

Related work

The section provides an overview of recent work on parameter-efficient fine-tuning techniques, categorizing them into five groups: adding adaptors, adding prompts, model pruning, partial training, and low-rank decomposition.

Adding adaptors involves introducing trainable modules, called adaptors, into the original frozen model and training only the adaptors. Examples include injecting adapters between layers or adding sparse, task-specific difference vectors.

Adding prompts involves prepending new tokens to the input text and training only the embeddings of these prompt tokens, forcing changes to apply to the corresponding vectors while keeping the core model frozen.

Model pruning removes certain weights from the network based on criteria like low magnitude or input activation norms.

Partial training involves training only a subset of the model, such as the final layers, bias parameters, or the most important parameters determined by Fisher information. The proposed method falls into this category.

Low-rank decomposition methods approximate model updates during fine-tuning using low-rank decomposition, training only low-rank matrices instead of the whole model. Techniques like Low-Rank Adaptation (LoRA) decompose weight changes into low-rank matrices or separate magnitude and direction components.

Conclusions and future work

The paper examines the components of BERT when fine-tuned for various GLUE tasks. It demonstrates that LayerNorm undergoes more changes after fine-tuning compared to other components. This finding aligns with previous research by Kovaleva et al. (2021), which showed that disabling LayerNorm significantly harms BERT's performance.

The study reveals that only fine-tuning LayerNorm achieves comparable performance to Bitfit, a method proposed by Zaken et al. (2021), despite being more sparse. Using Fisher Information, the researchers identify important subsets of LayerNorm parameters. They demonstrate that by fine-tuning as little as 10% of LayerNorm parameters, which is a tiny fraction of the BERT model, similar results can be obtained with only slight performance degradation.

The paper focuses on layer normalization, a popular normalization method in NLP. It suggests extending the parameter-efficient training approach to batch normalization, widely used in computer vision, to improve computational efficiency for training batch normalization models.

Appendix A Normalization in neural networks

This section discusses batch normalization, its shortcomings, and layer normalization as an alternative solution.

Batch normalization normalizes inputs across a mini-batch to reduce internal covariate shift during training. However, it has several drawbacks:

1. Dependency on mini-batch size, which can introduce variability.
2. Less effective in recurrent neural networks due to sequential data.
3. Complications during inference due to needing running averages.
4. Introduces batch dependency, affecting generalization.
5. Inability to maintain criticality, leading to gradient explosion.

Layer normalization normalizes across the layer instead of the batch, addressing batch normalization's shortcomings. It performs well in RNNs, avoids batch dependency, operates the same way during training and inference, and maintains criticality. Layer normalization computes the mean and variance across features in a layer, normalizes the inputs based on these statistics, then applies an affine transformation with learnable parameters. Variations have been proposed to improve gradient propagation. The technical details of layer normalization's computation are provided.

Appendix B Distance definitions

The L0 distance between two vectors V1 and V2 of size n is the number of elements where the values differ between the vectors. This is similar to the Hamming distance measure. The L1 distance, also known as the Manhattan distance, between two such vectors is calculated as the sum of the absolute differences between the corresponding elements of the vectors. A mathematical formula is provided to calculate the L1 distance.

Appendix C Metrics for GLUE results

The section does not provide enough information to summarize. The statement "Table 4 shows the metric used for each task" lacks context about the specific tasks, metrics, or research being referenced.